In Part 2 of Know Your Adversary’s celebration of 10 years of BloodHound, Jared Atkinson and Justin Kohler are joined once again by BloodHound creators Andy Robbins, Will Schroeder, and Rohan Vazarkar. Picking up where Part 1 left off, the group explores how lessons from the original BloodHound and the Active Directory Adversary Resilience Assessment (ADARA) helped turn attack path analysis into a methodology defenders could operationalize, ultimately laying the groundwork for BloodHound Enterprise.
The conversation dives into a fundamental challenge they discovered along the way: defenders can’t simply enumerate and fix every attack path. Instead, they need to understand what’s wrong, how much it matters, and what to fix first. The group discusses how this realization shaped BloodHound’s approach to prioritization, tier-zero isolation, continuous monitoring, and giving defenders the context they need to reduce attack path risk as identity environments constantly change.
From there, they trace BloodHound’s expansion beyond on-premises Active Directory into Azure, hybrid identity environments, GitHub, and other platforms. They share the technical and engineering challenges of modeling increasingly complex systems at enterprise scale, how experimentation with GitHub helped accelerate the development of OpenGraph, and why extensibility became essential to BloodHound’s future. The episode closes with a look toward what comes next, including new attack graph coverage and the continued expansion of BloodHound into emerging identity systems.
00:00:12:02 - 00:00:12:17
Jared Atkinson
Welcome.
00:00:12:19 - 00:00:41:01
Jared Atkinson
Welcome back to the Know Your Adversary podcast. I'm Jared Atkinson and my co-host Justin Kohler. We're joined again. If you listen to the previous episode by Andy Robins, Rohan and Will Schroeder. Those are the the creators of BloodHound. And in the previous previous podcast episode, we talked about kind of like the origin of BloodHound, where the idea came from, kind of a lot of fun stories about the history and kind of like the different challenges that they sought to overcome, and we ended up running out of time.
00:00:41:01 - 00:01:01:14
Jared Atkinson
So then we realized that if we want to talk about kind of continue talking about how it evolved into, for instance, BloodHound Enterprise and where it's going to go in the future, how we kind of all see that as a as a collective. We realized we needed another podcast. So welcome back guys. Nice to see you again. Especially that mustache will.
00:01:01:16 - 00:01:37:18
Jared Atkinson
So in the in the previous episode, we, we chatted about kind of this thing that we called Adara, that Rohan and Andy you had been conducting for some of our customers. And this was basically using the original OG BloodHound, what we now affectionately called BloodHound Legacy, to try to kind of like flip the script and look at look at the problem from the defensive perspective of how do companies leverage this information that, like the attacker, attack path view and actually like make their security controls more robust or like reduce, reduce the amount of attack paths that they have to face and do that in a systematic way.
00:01:37:19 - 00:02:12:12
Jared Atkinson
Maybe you guys could touch on a what? Maybe if you recall what Adara stands for, I don't want to. We didn't talk about this beforehand. I don't want to put you on the spot, but if you don't recall, skip that and then go into kind of like what the generalized idea was, maybe what some of the struggles that you ran into from that perspective were, and how that kind of like evolved into BloodHound Enterprise back in 2016, when Will Rohan and I initially released BloodHound, I remember there was a moment when we were at the Blackhat Arsenal booth showing our little five minute demo of BloodHound over and over and over again, and I think
00:02:12:13 - 00:02:41:05
Andy Robbins
probably the number one most common question we got was how much does this cost? Which of course it was free and open source. But even back then, back in 2016, we knew and I think it was pretty obvious to anybody paying attention, that the really compelling value of the product was on the defensive side. Now, personally, my own opinions here are that like on the offensive side, red teams really don't like spending money on tools.
00:02:41:08 - 00:03:06:07
Andy Robbins
There are some exceptions to that, obviously. And then kind of on the defensive side, it's it's maybe a little bit the the opposite. But I would I would let Justin speak to to all the business side of all that because he's, he knows that a lot better than I do. I think that's objectively true though is that the just the like total addressable market and the funding for red teams is fairly, fairly small and it's mostly centralized in people.
00:03:06:08 - 00:03:26:03
Jared Atkinson
Right? So like they spend money on on people that have capabilities. And then a lot of those people are programmers and developers and have those types of abilities to where they can take something that is an open source capability and maybe extend it a little bit or, you know, turn it into something that's repeatable. And so there's there's just kind of like a different vibe on that side.
00:03:26:04 - 00:03:28:16
Justin Kohler
It's also extremely utilitarian, right?
00:03:28:17 - 00:03:47:00
Justin Kohler
So like, like the the types of issues that you're going to find with BloodHound can be executed by red tumor, but to fix them they may need to be presented in a completely different way. And that's another layer of complexity on top of any product. And that's where like, you know, defensive people would step in.
00:03:47:01 - 00:04:06:05
Jared Atkinson
Not to keep not to keep belaboring this. But there's also a but I will, I guess. But there's there's also like a scale problem from the red team perspective. And then go into the defensive perspective. And like this is the the defenders dilemma I guess, which there are obviously counterpoints to that. But like kind of the idea that the attacker only has to be right once and the defender has to be right all the time.
00:04:06:05 - 00:04:17:14
Jared Atkinson
And so there's there's a scalability and like operationalization aspect of kind of like what Justin's talking about, where you need to get tons of different parts of the business involved. All right. Please continue. That was.
00:04:17:14 - 00:04:55:13
Andy Robbins
A big thing was was operationalization of the technology for defensive purposes. That was that was something that we heard a lot from a lot of our consulting clients was help us operationalize this thing for the defensive side, one of the problems that we faced was that BloodHound, released in 2016, was made by us, and it was made for people who were doing the work that we were doing at that time internal Active Directory, penthouse, Red team assessment, that kind of thing.
00:04:55:15 - 00:05:27:04
Andy Robbins
So the software was architected, built, and let's say quote unquote operationalized just for red teamers and pen testers, not for defenders. So when it came time to finally say, all right, we're going to really take a shot, we're really going to like, try to make something that the market will actually adopt and more importantly, use, because software that doesn't get used really doesn't have much value.
00:05:27:06 - 00:05:57:14
Andy Robbins
And we wanted the product to create value for those customers by actually helping them to solve the problem of attack pads. That's easier said than done. And I remember getting some really good advice very early on from one of our company leaders that instead of trying to just start building something that, you know, we think, okay, we got this great idea, let's go ahead and do it.
00:05:57:16 - 00:06:16:09
Andy Robbins
This person's recommendation was create a methodology first and then have the product automate that methodology. That was the genesis of the Adara, which stands for Active Directory Adversary resilience assessment.
00:06:16:10 - 00:06:17:16
Jared Atkinson
Resilience. Did you set.
00:06:17:16 - 00:06:23:20
Justin Kohler
A record for longest service name when that came out? By the way.
00:06:23:22 - 00:06:46:14
Andy Robbins
I'm sure McKinsey has this beat, you know, somewhere there. But at least for internal services, I think I think it might be the longest name that this company offers. Yeah. So the Adara service, our primary objective there was to create value for our Adara clients by extracting defensive value out of the graph. Me and Rohan doing that personally.
00:06:46:17 - 00:07:04:00
Andy Robbins
And as we were doing that, we started to learn a lot of really hard lessons. For example, you cannot see all attack paths, you cannot read all attack paths, you cannot load all attack paths into memory.
00:07:04:01 - 00:07:05:00
Jared Atkinson
You can't even fathom all.
00:07:05:02 - 00:07:33:16
Andy Robbins
Can't even fathom all the attack paths. There was an analogy that still today is the most useful analogy for me, which is a map. Just imagine Google Maps. Imagine a map of of North America, and we're talking about, you know, what are the attack paths that get us to domain admin or to some kind of red team objective, and instead think about like, well, what if what if I'm in Los Angeles and I want to know a route to get to to New York City?
00:07:33:18 - 00:07:55:08
Andy Robbins
Well, there's countless, countless routes that will get you there. You can go on this interstate, that interstate. You can take a flight, you can take a train, you can go on a bike. You could walk. So if you tried to just enumerate every single path there is and every mode of transportation there is to take between Los Angeles and New York City, you can imagine that list would be pretty long.
00:07:55:08 - 00:08:25:01
Andy Robbins
So just imagine trying to list out every single attack path that connects from domain users to domain admins. This is the lesson that Rohan and I learned, where we would start running a cipher query against the BloodHound database and come back maybe eight hours later, maybe nine hours later, running some kind of experiment, or trying to get some kind of intelligence out of the graph in this way, 8 or 9 hours later, if the query didn't crash, then we would get some data out of it.
00:08:25:01 - 00:09:06:08
Andy Robbins
And then, you know, doing that enough times, we kind of realize, like, we're not really getting anywhere. We're not we're not really giving anything to our customers that says this right here is the issue to fix right here. That changed once we had a pretty stark realization about how to combine the problems that Active Directory admins faced with securing Active Directory with the graph technology, we figured out what combination we figured out how to connect those two issues with a solution that that actually does, in my opinion.
00:09:06:13 - 00:09:43:18
Andy Robbins
And I think the evidence shows this as well. Reduce attack path risk in these client organizations. And really the crux of that, of that realization was that we don't need to know what all the attack paths are. Now, the graph knows, and the information about those attack paths is in the graph. But I as a defender, if I put myself in the shoes of Active Directory administrator, really the the thing that I want to know is what's wrong, how bad is that, and what do I need to do to fix it?
00:09:43:20 - 00:10:21:01
Andy Robbins
Brass tacks. I want to do my job and I want to do it well. And so I need to know from this product, like help me understand what's wrong. How bad it is and how to fix it. We focused in on the concept of tiered administration and then to a secondary degree also the concept of least privilege. And we found a way to kind of use the product to connect from all of the attack path data through this Adara methodology into outputs.
00:10:21:06 - 00:10:38:23
Andy Robbins
In our reports that gave those Active Directory admins the answers that they needed to these Active Directory security questions, that without the graph are, in my opinion, really not possible practical to answer.
00:10:38:23 - 00:10:39:18
Justin Kohler
So before.
00:10:39:18 - 00:11:06:08
Justin Kohler
You were you were just kind of like throwing queries, getting really frustrated if they if they returned data, great. But then you just have to ask more questions. Or if they didn't then you'd have to choose a different route. I remember talking to you guys back then. That's when we kind of got introduced. And you like this is a long process, like you're talking like six weeks of just like data collection and continuing to spend, and then you dump this, like, report.
00:11:06:10 - 00:11:24:18
Justin Kohler
And now this isn't the isolation which is in the lead into BloodHound Enterprise, but these report would just have amount of information in it. And it was all critical information. It's like to back to your point, well where do I start or how bad is this right. It's bad. But how bad is this you guys got.
00:11:24:20 - 00:11:26:23
Andy Robbins
There were a few things. Just jump in.
00:11:27:03 - 00:11:28:16
Justin Kohler
Yeah, yeah, yeah, yeah.
00:11:28:17 - 00:11:52:19
Andy Robbins
There were a few issues where, you know, we would get some answer out of the graph, and it's kind of inarguable, like if this is if this matters or not, or if this is this should be prioritized or not. I remember one day I interrupted a family vacation that Justin was on. He was actually on the beach, as I recall, and I wanted to call you up because you had a better contact with the with the with the client than I did.
00:11:52:21 - 00:12:09:22
Andy Robbins
And we had found something that was very obviously bad and rang you up and like, yeah, man, I'm on the beach right now with my with my kids and my family. Can you like, not bother me right now? But like, you know, an issue might come out of the graph where it's like, you know, let's.
00:12:10:00 - 00:12:16:07
Justin Kohler
Do clear. We ended up giving you the contact to that customer, though. It didn't just leave you hanging.
00:12:16:09 - 00:12:36:23
Andy Robbins
Yeah, yeah yeah, yeah. And the customer like it. Yeah yeah yeah yeah yeah. So, you know, we have issues that would come out of the graph where like, it's obvious, like there is no discussion that has to happen about how is this how bad is this configuration. Actually, let's say for example, the domain users has been added to the domain admins group.
00:12:37:00 - 00:12:59:23
Andy Robbins
You know, that group has been added to the other group. Okay. That's there's no there's no real discussion that has to happen. Like that's that's not subtle. That's pretty obvious. But like you said, Justin, like once we would get past kind of those low hanging fruit items, which to be clear, the graph really didn't uniquely answer. You can get those answers from other things.
00:12:59:23 - 00:13:24:06
Andy Robbins
Where where the where the graph really shined was in kind of all the other issues that seemed maybe not that bad. So let's say maybe there's a domain admin that logged on to some random server somewhere. Okay, that happens all the time. No big deal. Who cares? But then with the graph uniquely enables is figuring out, okay, why does that matter?
00:13:24:06 - 00:13:46:18
Andy Robbins
How bad is that actually? So the domain admin logged on to a server. Okay. That now means that that domain admins session is on that system. It could be lifted by an adversary, it could be reused by an adversary. So what the graph enables us to do is kind of start at the end and go backwards. So instead of LA to New York, let's start in New York and let's start tracing our step steps backwards.
00:13:46:18 - 00:14:23:21
Andy Robbins
So let's say we start a domain admins and we find oh. Guess what domain users has been added to the domain admins group. Okay, that's pretty obvious. Let's shut that down. But then we find okay, here's domain admins. There's a user who's in the domain admins group. That user logged on to a system. That system has this user with local admin rights on it, that user logged on to this system, etc., etc. and what would happen a lot of times is and this is getting back to what you were saying, Justin, is what the graph really uniquely enabled us to do was to put a so what measurement on each of those steps in the
00:14:23:21 - 00:14:47:15
Andy Robbins
attack path, including the most important ones, which are the ones that end at compromising something in tier zero, which is my opinion. That was really kind of like one of the big insights from the Adara service line that that led us into what eventually did become BHE, which was okay. We don't want to try to compete with like a whole bunch of other things that already exist in the market that do all this stuff already.
00:14:47:17 - 00:14:56:17
Andy Robbins
What can what can this thing do that is unique to it? And let's focus on that. And that was that was one of the key insights that we that we had during that, during that time.
00:14:56:17 - 00:15:23:00
Jared Atkinson
I've started to think of this as like localized intelligence. So you have you have like global intelligence, which is, hey, if you already pee into a computer, you've exposed your credentials to anybody to be able to dump them. Right? That's like that's just a thing that we know the localized intelligence is saying, despite the fact that you're doing the same behavior several times the the net outcome of that behavior or the potential outcome of that behavior is not equivalent across all those instances.
00:15:23:00 - 00:15:42:11
Jared Atkinson
Right? It matters what the context is that, that that that behavior kind of like connected to, which is your point. Like you could, you could RDP as an admin into two servers and you could have wildly different risk profiles that are associated with those based on the basically inbound exposure of that server that you're that you're already paying into.
00:15:42:11 - 00:15:47:06
Jared Atkinson
And I think that's like that's a massive kind of like change in how people think about the problem.
00:15:47:07 - 00:16:07:17
Justin Kohler
Yeah. So like there's now we had this like methodology. Right. And this is when I came on board and we were going to like okay, there's there's a clear use case. There was also other things that you found like again, a service or this kind of like this, this manual effort of finding these things is only good that literally the second you build it, because the graph is constantly changing.
00:16:07:18 - 00:16:29:05
Justin Kohler
Right? So like, you know, how would you verify if you fix something, how like what if somebody does something really, really terrible the next day and your priorities change? Well, you have no idea. Right. And so this like you guys also at the same time wanted to say, well, you really need a view of this at all times so that you can make sure that you're you're spending your time in the right effort because you can't you can't act on everything.
00:16:29:05 - 00:16:31:01
Justin Kohler
So the issue there is.
00:16:31:04 - 00:16:53:10
Andy Robbins
Is that these environments change. So people are onboarded into companies. That creates a new ad account. People move from department apartment to move out. They get new security group memberships. So over time all these things change. But that over time aspect is just one piece of it. The more important piece is that they're changing every hour of every day.
00:16:53:10 - 00:17:29:19
Andy Robbins
So the environments are not static. It's it's unlike the analogy of the map of North America where the interstates and the roads are pretty static. They don't move roads that often. Security group memberships and Active Directory ACLs, group policy, all that stuff is changing all the time. So that issue of the environment being constantly varied and the environment being in flux at all times, meant that the solution had to be able to keep up with those changes so that you're not waiting for another professional services engagement.
00:17:30:00 - 00:17:46:13
Andy Robbins
12 months later, the customer is like, really? What they need is they need, like I said before, like, what's wrong? How bad is it? How do I fix it? And also what's changed like day by day, like has any has any new risk come up or week by week or whatever? Not not year by year that that cadence is way too slow.
00:17:46:13 - 00:17:51:17
Andy Robbins
And so because of that cadence requirement, a product is really what makes the most sense.
00:17:51:18 - 00:18:12:12
Jared Atkinson
And that changes entropy essentially. Right. So it's like, I have a plan, I'm an architect, I set up things as I, as I think that it's most secure. But the problem is, is that the architecture was only only potentially true. Sometimes the architecture isn't even ever represented actually. Right. Because it's more complex than that. But it was only potentially true on day zero, right.
00:18:12:13 - 00:18:25:20
Jared Atkinson
And then after that, people take over and start doing all kinds of random things. And that just creates entropy in the system. And entropy in the system means that it devolves towards less security. Let's say that right. So like evolves towards more risk.
00:18:25:20 - 00:18:27:12
Andy Robbins
I would say so yeah.
00:18:27:14 - 00:18:57:02
Justin Kohler
Then we so we have this we have this like method. And we're going to we're going to develop this like continuous solution that eventually ended up coming out with BloodHound Enterprise. And those you mentioned the tier demonstration that now is represented by by zones. We don't you don't we don't have to you don't have to enforce a tiered administration model zones can be kind of abstracted or generically applied based on how you want to isolate, like sensitive resources internally, but that that existed from the start.
00:18:57:03 - 00:19:13:13
Justin Kohler
Right? This this idea of I want to protect this. And that goes back to the New York analogy. And funny enough, like early on, if you got a BloodHound or demo of BloodHound Enterprise, you saw myself and Andy owned a call and we would pull up Google Maps and we would literally tell that story, and that was how we demonstrated.
00:19:13:13 - 00:19:31:18
Justin Kohler
And then we would show you the product. It wasn't, you know, that's how we had to explain how this works. It was also funny. There was other things that you mentioned learning along the way. I remember learning a lot about how defenders needed to receive the information. So yes, what is it? How bad is it and how do I need to fix it?
00:19:31:19 - 00:19:54:08
Justin Kohler
How do I need to fix it? Got really, really important because the way that we were presenting data previously was this is an attack thing to remove it. And it's like, well how and explain that more. And it got into the what we wanted to do is like bridge the identity and the security or like the add at that time, the add architect or the add admin and the security teams.
00:19:54:09 - 00:20:13:00
Justin Kohler
Right. Because they were they're kind of lobbing things back and forth and it's like, no, no, no, you guys need to see this together. And speaking the same sheet of music because ad teams would have rightful like objections. Like you need to explain this further or how do I fix this? Or what bad could possibly happen from this?
00:20:13:01 - 00:20:18:00
Justin Kohler
So I remember that like really like early on it was like, oh yeah, it turns out a.
00:20:18:00 - 00:20:23:01
Rohan Vazarkar
Bunch of red teamers aren't the best. Explaining defensive concepts to defenders. Who knew?
00:20:23:05 - 00:20:33:13
Jared Atkinson
I think the entire the entire industry understands that, right? Like if you've ever had a red team on your company, you probably know that. Just fix it. Not my problem. I pointed it out to you.
00:20:33:15 - 00:20:58:00
Andy Robbins
Yeah. And I think I think maybe the shared history that a lot of us have with being consultants for big chunks of our careers. A lot of us learned this lesson pretty early on that just because the pen tester delivers a finding doesn't mean anything. The finding has to be useful. It has to be correct. It has to be defensible.
00:20:58:01 - 00:21:17:10
Andy Robbins
It has to be falsifiable. So with the product we knew going in that if the product is going to show a finding, the person who is going to be asked to remediate that finding, it's probably not the same person who initially sees that findings, probably not the same person who is using the product on a day by day basis.
00:21:17:14 - 00:21:41:00
Andy Robbins
The other the other thing that I think a lot of us learned from consulting in our careers was that there's really, in my opinion, not much use for blame in a situation where the risk is outweighing the necessity to figure out kind of root cause.
00:21:41:04 - 00:22:01:01
Justin Kohler
So you mentioned like like blame, does it? Like we don't blame doesn't need to be part of this conversation. And we I think we realize that because it's impossible to understand this unless you had something like BloodHound Enterprise. So like there was like everybody was doing the best that they could with the tools they had at their disposal.
00:22:01:01 - 00:22:18:16
Justin Kohler
And like, if you turn the lights on and you can finally see it, yeah, of course there's going to be decisions like made ten years ago by people who never, like, don't work there anymore and can't explain them. That's not important. What we need to figure out is how do we move forward and remove the risk because it's affecting the organization.
00:22:18:16 - 00:22:42:10
Justin Kohler
Now, if, if I may, I might share like a funny story from early on. I remember when we first launched and we had a customer that said, so we just went through this engagement with Microsoft at the time and we completely isolated. Like basically they had put in the concept of the privilege zone and and they were confident that there were no attack paths there.
00:22:42:11 - 00:22:45:02
Andy Robbins
This was something similar to Red forest.
00:22:45:03 - 00:23:00:08
Justin Kohler
Yeah, yeah. And so they wanted to make sure that they just put in a whole bunch of money. I mean, it took them a year and not only the time to do it, but the resources they spent on the engagement. And they wanted to protect that investment. And so they saw our product as a way to make sure that that isolation never changed.
00:23:00:11 - 00:23:16:08
Justin Kohler
In a minute. They saw an attack path form that it would remove and remember asking them. I was like, wait, so we'll deploy it and we won't show you anything. They're like, yeah, like, well, how would you test it? And they're like, well, we'll we'll actually put a village just change in and see if you see it. Well, it turns out there was a lot more that that was missed.
00:23:16:08 - 00:23:27:20
Justin Kohler
And again it goes back to that not blaming like they could not see it. They did a lot. They had less, but they definitely did not have zero. They were able to get you to zero once they saw it.
00:23:27:22 - 00:23:56:02
Andy Robbins
That was that was another good example of of an architect. In this instance, a third party architect from a different service provider came in to this mutual client of ours and architected something similar to a Red forest architecture for them. And personally, what I have seen from those setups historically, you know, is that the architecture is great, it's solid, it's it's well thought out.
00:23:56:04 - 00:24:21:07
Andy Robbins
It does effectively protect the most critical accounts in the organization. But then that architect leaves, the engagement is over. And what happens is like what Jared alluded to earlier is that you have changes that happen. You have business requirements that change, you have shadow it. You have privileges that compound with other privileges. You have this system that is dynamic and changing.
00:24:21:07 - 00:24:42:01
Andy Robbins
And to be, to be perfectly frank, a little hard to understand if you don't have all the arcane knowledge about how all the Kerberos stuff works, for example. So really, who could be blamed for kind of having a Red forest deployment go out of control? I think it's from my experience, it's what's happened with most of them.
00:24:42:04 - 00:25:02:10
Justin Kohler
Another funny story from from early on, I remember because we believe that like, there's no blame. And the only way that you can see it is through something like this. It's quite literally impossible. You were talking with an individual that were different company where they were saying, well, I don't know why I would need this because I've separated my admins from my users.
00:25:02:10 - 00:25:04:11
Justin Kohler
And Andy, you said.
00:25:04:13 - 00:25:08:15
Andy Robbins
I said the the thing that we had developed will be able to prove that what you just said is true.
00:25:08:17 - 00:25:10:04
Justin Kohler
Yeah.
00:25:10:06 - 00:25:13:00
Jared Atkinson
Was it true?
00:25:13:01 - 00:25:17:13
Andy Robbins
No comment.
00:25:17:15 - 00:25:30:15
Justin Kohler
So so we talked to a lot about like what? And if people want to learn more about BloodHound Enterprise and what it does like, you can you can go on our website and find all about that. So we don't need to go into like every feature and like why we did it. But that's what we were trying to like.
00:25:30:15 - 00:25:48:00
Justin Kohler
The problem we were trying to solve is get this in the hands of defensive teams, get it away and like prioritize and yada, yada, yada. Now then we moved into a different platform. So at the time it was just Active Directory. And then we tested on this little thing called Azure. Anybody want to take the first step of that one?
00:25:48:02 - 00:26:09:18
Will Schroeder
I would like to not take the first out of it that it's I still have nightmares about that implementation originally. So I'll prime the discussion of just saying, you know, this has been years and years ago, but it kind of emerged from a client that one of our workmates, Lee Christiansen and myself, were working on where he's like, okay, we're moving stuff into Azure.
00:26:09:20 - 00:26:26:12
Will Schroeder
I have the feeling that, you know, we're probably making or we might be making some of the same mistakes that we made historically, and there's just this huge push, you know, years ago, this huge push is trying to get everything up here. And we just we don't know how to handle this. It's like we barely on the security side.
00:26:26:13 - 00:26:52:18
Will Schroeder
They were struggling to even just like understand how the entire system worked. And Lee and I spent several weeks just kind of doing a crash course for the first initial, like very poor version of the model. That is not what is the the accurate version of this and the product now and that kind of, you know, super kind of beta was still very interesting to him at the point of just being able to was kind of the first connection of some of those on prem graphs to crossing into the cloud.
00:26:52:19 - 00:27:14:03
Will Schroeder
I think. What is the term use? Jared is like the meta graph or the, the type, the graph of graphs. Right. You know, like where we started to realize at that time those kind of the first example of this story had all these different authentication subgraphs and these attack subgraphs. And then we were starting to realize like, oh, these can connect to like what the actual threat, threat territory or map is not the territory.
00:27:14:03 - 00:27:22:11
Will Schroeder
All that kind of stuff. Right? Jared of like this is actually what attackers are abusing. And that was kind of the first initial like, oh, this this could be a big deal.
00:27:22:13 - 00:27:49:21
Jared Atkinson
I think you're establishing a security dependency chain, right, to where in many, many of the configurations of AD and Azure AD is actually a security dependency of Azure, and maybe even vice versa, right? To where they're mutually dependent upon each other, to where control over either means control of the other. Just to have fun factoid I was working on some defensive stuff with the the same customer, and at the time he called you and Lee evil Geniuses, a little evil scientist for being able to figure figure all that out.
00:27:49:21 - 00:27:58:07
Jared Atkinson
So that was that was fun. This was like, I mean, nobody had any insight into like, what attacks were possible in Azure at the time. I feel like and not going.
00:27:58:07 - 00:27:59:23
Justin Kohler
Into like every part of Azure.
00:27:59:23 - 00:28:00:16
Justin Kohler
But we quickly.
00:28:00:16 - 00:28:21:10
Justin Kohler
Found out that like they made some improvements in learning from their mistakes and Active Directory, but there was enormous amounts of complexity. So like a DC could be represent the single source of truth for an Active Directory domain. But in Azure you had all these competing services. And again like skipping to the point later we'll talk about you see the same thing in other platforms.
00:28:21:10 - 00:28:51:06
Justin Kohler
So AWS for example, S3 has an entirely separate system governing the rights over S3. You know that isn't AWS IAM, so you just had to understand a lot more, I guess any like high level thoughts from anybody on in terms of that push into Azure. And I also want to say that like the other thing that we were finding from clients is they had all the legacy, like the push from for Microsoft at the time was let's migrate it up to Azure and then like, you don't have anything in Active Directory.
00:28:51:06 - 00:29:06:15
Justin Kohler
And we saw almost nobody do that. Right. And I think we still see almost nobody doing that. And so they maintain this hybrid connection. So you have all the risk of your legacy tech debt and all the new risk of the cloud. And people are just trying to make sense of that.
00:29:06:21 - 00:29:29:21
Andy Robbins
Yeah. What comes to mind for me from that time is, is, you know, similar to what Will was saying with coming from, you know, Active Directory experience and then coming into a new platform that, you know, I personally had never really interacted with before. I saw a lot of the same kind of attack path primitives that you would expect to see in any IDP.
00:29:30:01 - 00:29:51:23
Andy Robbins
So, you know, a user can change someone else's password. Of course, that works very differently in intro than it does with on prem add. Somebody could add someone to a security group. Groups can have role assignments and stuff like that. So I saw a lot of the same kind of design decisions in intra and Azure that can lead to the emergence of attack paths.
00:29:51:23 - 00:30:32:09
Andy Robbins
So, you know, there are attack paths there. But what I also remember was noticing some of the design decisions Microsoft, I believe, made that I think actually prevented the emergence of a lot of really critical attack paths. And I think the best one that I can think of, the most obvious one I can think of, is with Active Directory, you have these kind of all inclusive security groups or principles like domain users, everyone, authenticated users, and it's common to see in real environments those principles be given group memberships or privileges, which then means that everybody has those privileges.
00:30:32:09 - 00:30:59:23
Andy Robbins
And the difference that I saw with Intra was that there there was no all inclusive security group, especially a default one that everybody already belongs to. That, I think, was a great decision on Microsoft's part to not include something like that. I think that prevented the emergence of a lot of really critical attack paths within an intra tenant itself.
00:31:00:02 - 00:31:21:00
Justin Kohler
And so the I think the other thing that we found, and we see this now a lot as we're going to different platforms, is the early on, the Active Directory administrator may or that team could be learning about Azure. But over time it was completely separate teams. And so you'd have decisions made by either one of those teams that would affect the other.
00:31:21:00 - 00:31:43:16
Justin Kohler
But unless you were able to see it, you couldn't you wouldn't understand that. I remember early on, like when we first connected the first hybrid attack pads, which was the synced user everywhere in Microsoft's documentation tells you not to do this. And when we did this, like when we showed it, every single customer that we see was doing it.
00:31:43:17 - 00:31:46:05
Justin Kohler
And I think the reason it.
00:31:46:05 - 00:31:46:13
Justin Kohler
Made it.
00:31:46:13 - 00:31:55:12
Andy Robbins
And saying that thing, that thing to be clear that they were doing was they were synchronizing their tier zero accounts back and forth between the two different platforms. Yeah.
00:31:55:13 - 00:32:18:23
Justin Kohler
Which is a big no no. So any privilege roles should not be synced. And we saw it everywhere. And I think part of it is you couldn't it was an easy thing for people to do. And but it affected the overall system. And unless you can see that like you're not aware. Right. And nobody's doing necessarily anything bad, they're trying to get people to access the resources that they needed to access.
00:32:19:00 - 00:32:54:10
Justin Kohler
I kind of want to we, we, we expanded within Azure and Active Directory considerably over that time. So shortly after Azure was then ADC's. So that was based off the enormously popular research in Active Directory certificate services by Will Schroeder and Lee Christensen. Early Christensen. The then we went to so the continued expansion within it within Active Directory in Azure always always happens however at the time will and Jared you you were trying to go in a different route entirely.
00:32:54:10 - 00:32:59:17
Justin Kohler
So outside of the Microsoft, well, kind of outside of the Microsoft, it's because they acquired GitHub.
00:32:59:18 - 00:33:19:15
Jared Atkinson
I think maybe I'll, I'll kick it off. So I, I run the research team. So we're always kind of like trying to figure out what should we what should we look at next. And one of the things that was interesting to me was GitHub. And the reason why this was interesting, not because I had any particular insight, but because we had had several red team requests to kind of look at the Cicd, kind of like pipeline.
00:33:19:15 - 00:33:40:01
Jared Atkinson
And I know GitHub is only one component of that, but it's kind of like a central component. And I think we'll we'll and Lee also had a similar engagement where they were, they were asked to kind of like dig into GitHub configuration. So they had some, some like existing research that they had done on how you can gain access, how you bypass branch protections and things of that nature.
00:33:40:01 - 00:33:55:16
Jared Atkinson
And so we thought, hey, we'll just go ahead and throw this thing into BloodHound and we'll see what it looks like once we once we get done with that. And maybe Will was the one that actually threw it into BloodHound. I don't know if throw is the right the right verb, but he did something and maybe there was.
00:33:55:17 - 00:34:00:00
Jared Atkinson
There were some struggles that kind of led to new features.
00:34:00:02 - 00:34:27:21
Will Schroeder
So I'm not a BloodHound developer, you know, I helped write the original injector with the terrible PowerShell stuff back in the day. So for this, when I was working with Jared, it was and this was pre coding agent. So this had to be done entirely by hand. And it basically yeah. Yeah. Trying to do a branch essentially a private branch of BloodHound that could handle like manually grafting in the GitHub schema and all that kind of stuff.
00:34:27:23 - 00:34:46:03
Will Schroeder
So it was we realized it was a challenge. Also part of it was we were trying to see like, could someone that's not on the development team do this, you know, without having a hand there, having to have their hand held the entire time. So we achieved it. It was really interesting. But we realized that that wasn't sustainable.
00:34:46:04 - 00:35:11:15
Will Schroeder
That's not something that, again, I wasn't going through like it was a proof of concept to see if this would even work or not. And I know that's I think that probably wasn't the inspiration, but I think that probably helped accelerate, you know, the concept that became open graph, right, Jared, of saying, like, we need a way to like from a research side or, you know, for like a client say they have their own internal system or private system or something.
00:35:11:15 - 00:35:32:18
Will Schroeder
We wanted to grant people the ability to expand the schema, at least, you know, even temporarily. Right. Or something. That's not necessarily something that's going to land into the main branch, like every single time. Sure, but have have an extension system that lets people model additional access control systems and attack paths. And I know GitHub was one of the first ones that you built for that, right?
00:35:32:18 - 00:35:33:06
Will Schroeder
Jared.
00:35:33:08 - 00:35:56:14
Jared Atkinson
Yeah. And I think we just for context, we'll probably worked on that for like four months or maybe, maybe even longer to try to get it into BloodHound and, you know, BloodHound when it first created, I don't think anybody was thinking that it would even go beyond Add. And so there's a lot of kind of like design decisions within, within the product itself that was, you know, kind of like on the, on the order of hard coded as opposed to choosing to be extensible.
00:35:56:16 - 00:36:14:04
Jared Atkinson
And that's just because it was what it was. Right. And so Will. Will was fighting against kind of those design decisions, and it ended up taking a long time. And I think we we did a presentation at Socan. Was it last in 2025 or was it in 2024 that we 20.
00:36:14:06 - 00:36:21:15
Will Schroeder
I think it might have been 20. No. Because open graph cap release in 2025. Right. So that's been.
00:36:21:17 - 00:36:22:21
Jared Atkinson
In 24.
00:36:22:23 - 00:36:24:03
Will Schroeder
I think so yeah okay.
00:36:24:04 - 00:36:43:05
Jared Atkinson
Yeah. So we did it. We we gave the presentation. Then we realized, hey this is not sustainable for what we want to do going forward. And so then the product team implemented Open Graph, which is this extensible framework for adding literally whatever you want into, into the BloodHound graph. And then we were able to finally get that thing released.
00:36:43:05 - 00:37:02:12
Jared Atkinson
We released the GitHub open graph extension in 2025. I don't even know what year it is, what? It's 2026 right now. So a year later at the at the subsequent whatever, whatever it is, the subsequent Socan we released, we released the GitHub open graph extension. And so it took us a year to kind of like get all the way through all those hoops.
00:37:02:12 - 00:37:09:08
Jared Atkinson
But now it's like Open Graph has just taken off, like selling like hotcakes. You might say.
00:37:09:10 - 00:37:42:23
Rohan Vazarkar
I just wanted to like, you know, I wanted to touch on one other thing that kind of came out of the whole process. A lot of the Da assessments was Andy and I sitting at a client site, like literally just hammering their network over and over again with data collection. And, you know, one of the I think more important things we learned during that, along with, like the idea of like tears or isolation was a scale for anybody else who ever wants to kind of follow in this footstep of like going from a consulting product to a real product.
00:37:43:00 - 00:38:13:00
Rohan Vazarkar
I will tell you right now that you are going to have no understanding of your scale until you actually run something like this. The time we spent on those client sites was basically us hitting the network, and me fixing bugs that I'm discovering in new networks that we've never hit before. I remember one that was like three weeks straight of Andy and I just sitting there trying to fix bugs in this like behemoth corporation that had like, the weirdest ad configurations you can imagine.
00:38:13:01 - 00:38:31:20
Rohan Vazarkar
And I don't think we ever even fixed all of the bugs in pound until like near the end of it. And it was just stuff we never run into or had to deal with. Because again, we built this for red teams and red teams don't care to get all the data, we just care to get like enough to get our attack out the domain admin or wherever else we need to go.
00:38:31:21 - 00:39:04:05
Rohan Vazarkar
Right? So, you know, it was a it was a totally different ball game doing this from the like, defensive perspective and understanding that like, we would have to collect pretty much everything in order to have the full picture and the full picture matters so much more on the defensive side that doesn't on the offensive side. And, you know, obviously hitting environments that we have never seen the size of before taught us a lot about just how we had to optimize things, how we had to scale our cipher.
00:39:04:07 - 00:39:31:21
Rohan Vazarkar
But part of the understanding of we can't hit all the tac paths was really driven home when we hit an environment that had five domains that were each like, you know, hundreds of thousands of computers apiece with like millions of users. It's like, oh, you know, this this might have worked on some of the smaller environments that we just kind of like yo load on, and then we just hit like a real environment where like, oh, all of our assumptions about what we thought we could do were completely wrong.
00:39:31:21 - 00:39:36:17
Rohan Vazarkar
Because if we're going to do this in real enterprise environments, this is what we have to deal with.
00:39:36:21 - 00:39:53:18
Jared Atkinson
An interesting kind of take away from that. A lot of times when we talk about scale, we think about size, which obviously you just you just gave that example and like the size of things does does matter. But it's also like the diversity. Right. And so it's the difference between works on my computer and works on everybody's computer all the time everywhere.
00:39:53:19 - 00:40:17:04
Jared Atkinson
And that's, that's more complex, like a kind of fun anecdote to go with. That is when we did when we built GitHub, a GitHub collector, we ran, we ran it on like the spectrum GitHub, and we ran it on some other small ones. And then we ran into a customer that had 50,000 repositories. And GitHub has very draconian rate limitations on hitting their API, because it's GitHub and everybody that interacts with GitHub is a developer.
00:40:17:04 - 00:40:36:20
Jared Atkinson
And so you can just imagine the types of abuse that people are, which maybe we're participating in, but other people are also participating in. And so they may they made it to where you could only make like 5000 requests a minute or an hour. Sorry. But the problem is, is that when you make when you have to query 50,000 repositories, that obviously takes ten x what the hourly rate limit is.
00:40:36:21 - 00:40:53:10
Jared Atkinson
And then there's additional subsequent queries that you need to make. And so it's just like this complete hot mess of what you're trying to trying to do. And you very quickly realize, okay, this naive way that I approach this originally is not going to be satisfactory. And you got to completely adjust everything.
00:40:53:12 - 00:40:55:00
Rohan Vazarkar
Product engineering is hard.
00:40:55:01 - 00:40:56:16
Jared Atkinson
Yeah. There you go.
00:40:56:18 - 00:41:13:21
Justin Kohler
Yeah. I mean, like at this point we've seen organizations with hundreds of domains. We have I can't be specific, but I'll just say that hundreds, multiple hundreds, not not 2 or 3, but more. And then like domains with.
00:41:13:23 - 00:41:34:13
Justin Kohler
4 million users in one domain, we saw one with like over ten. Actually we have one client has over 50 million users in one domain. I mean, the amount of stuff that you see, and that's again repeated across Azure, they're seeing it in very interesting, like new companies with their use of AWS, like the different ways that people use this platform.
00:41:34:13 - 00:41:55:02
Justin Kohler
It's crazy once you get into it and the types of scaling challenges you have. I also want to say that like it's one thing that handling the scale is, is one thing that's like just kind of a gate. You must clear. The other thing is can you get the data again, just like, can we make the data useful for an Active Directory administrator?
00:41:55:02 - 00:42:11:09
Justin Kohler
We have to make it useful. Now for a Jamf administrator or a GitHub administrator, or an AWS administrator or or or or or. Right. Because this awesome picture that we can show is, is not really useful if we can't show people how to fix it and where to fix it.
00:42:11:10 - 00:42:15:17
Jared Atkinson
Andy, what's your what you're saying? It's like if your product allows somebody to do it.
00:42:15:19 - 00:42:45:10
Andy Robbins
Yeah. What I, what I, what I've thought about like after seeing so many just things in Active Directory environments that make you wonder, why would anybody in their right mind ever do this, that, that. I think that was actually the wrong question to be asking. I think the right question, or maybe, maybe the right assertion or the right takeaway from that was that if a if a platform allows an admin to do something, at least one admin out there in the world is gonna do it.
00:42:45:14 - 00:43:00:05
Andy Robbins
Yep. So we've seen all kinds of different crazy ways that the ad schema can be extended. I mean, forget about like emojis in Active Directory user description fields, like the kinds of things that we've seen are they just boggle the mind.
00:43:00:08 - 00:43:12:12
Jared Atkinson
Yeah. Nobody, nobody, nobody thought when they were building an active Directory that they would have 50, 50 million users in a single domain, I would imagine. And but they put no limit on it, I guess. And so it's just gonna keep going.
00:43:12:14 - 00:43:30:10
Will Schroeder
Yeah. I mean, it comes down to write all these engineers or there's like they need to get their job done. So they're going to find creative ways to work around this stuff, just like we have for different problems on our side. Right. So it's I, I feel like I actually empathize more along of the time has gone on because you can look at something and say, this is like a mess or a disaster.
00:43:30:10 - 00:43:48:16
Will Schroeder
But I think now that we've been in this space longer, you kind of realize there's a reason people didn't just choose to make it super complicated or add all those users or whatever, or, you know, just because, right. Like there's there was a reason behind it, whether or not that was, you know, the best decision or not is a is a different story.
00:43:48:18 - 00:43:56:04
Andy Robbins
There's like a liberty and a power that comes from not worrying anymore about whose fault something was and just figuring out how to.
00:43:56:05 - 00:43:56:13
Andy Robbins
Fix.
00:43:56:13 - 00:43:59:17
Andy Robbins
It and worrying more about that.
00:43:59:19 - 00:44:01:08
Justin Kohler
Well, that, I think, is a great.
00:44:01:08 - 00:44:01:18
Justin Kohler
Place.
00:44:01:18 - 00:44:28:20
Justin Kohler
To stop again. It's been awesome to have you guys here ten years of BloodHound, and we're really excited about what we're announcing. If you're going to join us at Blackhat, come see us or watch the releases and then come ask us questions again during the BloodHound Slack, ask any questions you want. And as always, if you've never tried to try BloodHound Community Edition, run it in your environment, get permission first, and then see the problem like we do.
00:44:29:00 - 00:44:52:07
Justin Kohler
There's also some really interesting future conversations that are going to be seated here. There's a lot of work that you're going to see us announce a black hat, stuff like coverage for Microsoft Agent ID, which is kind of the first AI native view within the attack graph, which we're really excited to demonstrate. So yeah, more to come. Always a pleasure to to have folks on and talk about this stuff.
00:44:52:08 - 00:44:55:03
Justin Kohler
Jared or guys anything else you want to add?
00:44:55:04 - 00:44:58:01
Jared Atkinson
Oh good. Thanks again everybody. Oh thanks for having us.
00:44:58:04 - 00:45:00:02
Will Schroeder
Awesome. Thank you.
00:45:00:04 - 00:45:02:04
Jared Atkinson
Cheers. Thanks everybody. We'll see you next time.