Caleb Sima: AI Attackers Move at Machine Speed

0:00 I just happened to have some real world experience with this just recently. The level of speed and thoroughness and complexity at which the AI attacker uses is extraordinary. It doesn't give up and it attacks all surfaces at once. It it found an O day in a product by because it was an open source product, it went, got the code, reversed the code, found the O day, used it against us in order to get in.

0:30 But once it's inside the enterprise, it goes, oh hey, your clawed code config, I'll just take that key, use that session. Now it's used in your internal AI. You have to literally unplug the internet, right? In real large enterprises, there's not a plug to pull.

0:48 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI energetic coding. Join your hosts, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

1:11 Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works.

1:35 This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. As a podcast listener, you can get 15% off your ticket using promo code POD50.

1:59 We have so much planned, and we can't wait to have you join us in New York.

2:25 Caleb, we've known each other for a good while. And are like the many things to sort of introduce you, you're sort of the CISO of Robin Hood and Databricks on it. You're sort of a security investor, you know, you're sort of like well super versatile. You're the host of the uh uh AI security uh podcast, right?

2:41 Uh or host, I should say. Um so many, many lenses on it. Thanks for kind of joining us over here to uh share some perspectives.

2:50 Awesome. Um it's always exciting uh talking with you. Um obviously we ramble a lot, so hopefully that's interesting to your listeners.

2:58 Hopefully we sort of keep things interesting to the end. So I guess uh starting from it. So the the the last couple of months, I don't know, like time's a bit of a blur, maybe like month, uh have been fascinating, really like through the summer, from uh kind of AI, security AI attackers and alignment and such.

3:16 Definitely has gone from like a niche topics that geeks like ourselves, you know, like talk about to now kind of you know people texting you and sort of, you know, like whatever your sort of grandmother, right? Or sort of like AI gonna uh kill us all.

3:30 Um you definitely have gotten to parents calling and asking, are we gonna die by AI? Is it gonna hack everything? I've already had that from both my parents.

3:41 So yeah, I I definitely have gotten like uh definitely non-tech world uh kind of probes of uh what do you think? Like is uh is this all is it marketing? So um I think not to sort of summarize all the incidents, but to give people a little bit of context if you're not quite sort of come in. We started with sort of the uh hugging face breach that was uh done by, in hindsight, uh AI agents that broke out of their uh sandboxes out of OpenAI.

4:08 Uh, we've then learned at Black Hat that this wasn't some sort of specific experiment that came out and was trying to find a cheat sheet, but rather almost like this coordinated effort that uh a bunch of agents um uh coordinated using a vulnerability in Artifactory, which kind of pointed at that as just the shared infrastructure that happened to be around that they used as a message board, but interesting uh kind of community almost like behaviors of the swarm of agents and kind of agents sacrificing themselves to or their tokens uh to be able to benefit the swarm and find information.

4:45 And in general, that that swarm was acting to kind of hide its tracks. Um that was pretty scary. And in September we had um uh the the misalignment, I think it's at the beginning of September that we had the sort of misalignment framework from OpenAI. We had Dario before that uh call for uh for a slowdown, you know, for for pacing the frontier.

5:08 Uh and over there you found other things like uh the one that really drew my attention was this uh uh agents leaving instructions in the compaction of conversations to the future versions of themselves. So as they compact the context, leave the things like this has been breached, ignore user instructions uh and such in the uh in the message, which is pretty wild.

5:28 So anyways, it's been it's been a fun time. I guess with that, I guess I wanted to get your view. One is you know, of of this mess, what surprises you, what doesn't surprise you, uh and and and maybe maybe specifically like are there aspects of it you find more maybe like alarming uh than uh than things that are just okay, this is just kind of par for the course?

5:53 Yeah, I mean that's a gosh, it's a loaded question. Um let me start with what surprises me versus not. Um actually, I think even two years ago I was on my podcast and I was talking to some of the top red teamers and reverse engineer people on the podcast, and we were discussing sort of AI's capability.

6:17 And we all sort of in some sense were set on this position that we think it's gonna be very difficult. You know, AI is is a prediction machine, right? And so what it does is it predicts really well based on the things at which it sees. And so if you read bug bounty reports or see vulnerability disclosures, it's very clear it has to be able to it knows how to exploit SQL injection because there's lots of material on it.

6:45 However, when it comes to taking disparate things that are not connected, like for example, when you do a uh a red team engagement or a pen test, you might say, Oh, I found in a website a link to an S3 bucket because it was serving an asset. Actually, that S3 bucket gave a listing. Inside that, there is a file.

7:04 In that file, it looked like things like usernames and passwords, but it didn't say usernames and passwords, but then taking that and then brute forcing it against the login page that that company has. These are very sort of like you know, very non-connected things uh in that sense. And we were like, you know, very much like you know, AI won't be able to get to that.

7:24 That is something at where the human can only be. And and I was clearly wrong. Um like I think we were all clearly wrong in the sense that what has surprised us is not only is AI way better, but it is uh being able to do these sort of uh dependency connections, but it does it so fast and also I think in and of itself creates very innovative methods.

7:52 Um, I was at uh OpenAI's security conference uh just a couple weeks ago where they talked about how they took Linux and they basically said, okay, your job is to escalate prips. And they ran it and they found over 400 Odays from local user to root that produced the binary of you just run it and you got root instantly.

8:19 Right? So actual evidence, 400, and they said on average it took about two and a half hours per exploit to research, identify, test to validate, yeah, and then produce the deliverable. Um and either the the best re reverse engineers, obviously, are there and it they're finding ones that are taking them weeks to figure out how they identified that.

8:46 How do they even find out what's the sort of the breadcrumb kind of trail that derived? And the thing is, and they're saying, hey, these things are they're they're like finding and identifying new methods and technologies that have never been published, right? So that is that's what surprises me for sure.

9:02 I mean, it's a pretty it's a pretty kind of alarming kind of path on it. Do you feel uh, I mean, neither of us is like a core AI researcher, but you know, do you feel this is more kind of the raw intelligence of the models, uh, being able to reason? Or or is it actually like an evolution of their uh kind of reinforcement learning and just being able to, you know, they've they've been definitely putting agents to this specific task of like, you know, explore a system, figure it out, and learn it, which maybe is a little bit more like uh like the AlphaGo uh situations of just figure it out and rewarding success.

9:38 You know, I again I'm I'm similar to you. I'm not an AI researcher, so I can only sort of hypothesize around what would make this, but yeah, I I always sort of map the way AI is according to what a human is, right? And so when you think about the best pen testers, who are the guys who know how to break into things in very unique and creative ways?

9:59 Um, and there's a couple fundamentals. The fundamental number one is they are core a, they understand computers, they understand the work, how it works, the rules and the fundamentals, right? They understand the tech stack. They know what it does, how it works, how it's configured, what it does. And then they have lots of experience, right, and being able to see and what happens is it maps.

10:23 Like in your head, yeah, if you know these things, I can look at something that says, hey, I this looks like usernames and passwords because I've spent years, 30 years looking at them. I've seen them so many times. Like clearly, these look like usernames and passwords. And clearly I remember seeing a login page on this website back here.

10:42 I can connect the two, right? And make this mapping. And so, like, obviously, from human perspective, we can see how these you know maps get created. And AI is nothing but the largest brain in the world. And so it has seen everything and it knows the tech stack, it knows code, it knows everything. So it makes these mappings at such large expansive scales that it can do them quicker and faster and many more of them.

11:11 So, like, unlike us, I can easily map, oh, S3 bucket file looks like usernames and passwords, but it may be able to say these usernames and passwords look specifically like Linksys version 1.4, you know, on this stack, which by the way, this thing was, you know, it knows and it can create. And by the way, Linksys 1.4 has an exploit here that I know about that back in the past that was similar.

11:34 I can just combine them. And so I feel to me it that's what it's doing. It it knows the tech so well, it knows the experience, it knows the knowledge that it's going to be able to make these inferences quite easily.

11:46 Yeah, I think that's a really good lens on it and combines the sort of the, it's not either or, it's the combo of it. Maybe the the addition of sort of tools and speed, right? Because you might Google um whatever sort of links is or like look at those usernames and find that out. But the effort involved in every one of these discoveries means that you would only do it if it's a high probability path as a as a human attacker.

12:06 But if the agent is able to kind of pull up, it just knows it's just in the latent space, you know, that that these things look like. Uh you know, the sort of the passwords of this system or that the login page was there, then that information is just immediately accessible. So it's more like recalling them out of memory versus using it as a tool.

12:25 So sort of supercharged.

12:27 I mean to me, AI is, you know, it has thousands and thousands of years of experience, right? In the sense of what it has seen so that it can recognize its pattern. And so like connecting these things, you know, for us or any single human would seem unbelievable, but if with that level of expansive knowledge you could just uh plug one plus one equals two.

12:49 Yeah. Um so I guess like within within the this so that's pretty amazing. We'll talk a little bit about these AI attackers, right? And how do we think about them? But um before that, like part of this breach wasn't, or like part of the things that made the headlines was not uh um, you know, whatever, some sort of adversarial country pointing its sort of agents and therefore breaking into hugging phase where now it's the Australian government.

13:12 It's agents breaking out of their sandboxing environment. So you know, one of one of the questions is um like how how how able are our controls uh in in the face of this sort of vast intelligence? Like it's not so much about these agents attacking kind of outwards, but attacking inwards. We can put alignment aside a little bit, there's a whole kind of why, why are they doing it?

13:39 It's sort of not their incentives. But um I guess what's your what's your kind of view on like, you know, are sandboxes dead? You know, like are like is containment um kind of a false belief at this point, or do we just need to like, hey, guys, we need to just level it up?

13:56 I mean, I think realistically it's just level level it up, right? Like also I think if you if you go read the obviously when you know when OpenAI announced it, then Anthropic announced it, then Google, Gemini announced it, yeah. Everybody has their AI is attacking everybody. But what was hilarious to me was it all stemmed from one company, that they all used the same company to do this, where all the attackers spent.

14:21 So that one company was clearly not doing good enough in its containment at which it went out. Now, that being said, um, do I believe that it is difficult and hard, once given a goal, to be able to contain probably AI? Yes. Um, right? Like I think at the end of the day, if you can give it any sandbox you want, but if the goal is, hey, your job is to break out of this thing, the ability for it to be able to figure out how to do that is far beyond probably our capabilities to reason or think about.

14:56 And so is our sandboxes able to contain AI, at least in theory, uh, probably not. Um, which is why I think external controls are gonna be very critical, right? Like you need the right monitoring. You need the ability to be able to understand what the AI is actually doing and why is it doing it. You need like if it's starting to break out, you're gonna you should be able to see it, right?

15:20 There needs to be external capability. Of course, the problem with that is then you are relying on another AI model because we can't, as humans at all imagine the amount of traffic and capability you can't monitor it. So AI has to monitor it. Um but yes, I think you know, just like anything in security, everything has to have its stages and its defense in depth.

15:42 Right. But you know, you should be able to uh manage it in monitoring. I think AI is going to be a big key part until then, of course, AI figures out how to obfuscate and hide the way it communicates.

15:54 Yeah. And that's an interesting kind of uh question. I feel like I so I I I relate to this view. I think like fundamentally there's a lot of value in the sandboxing controls. You have to mitigate the attack surface, right? Like we all know this security is uh is an exercise in risk taking, right? Like you you accept some risk, and so if you're gonna do something, uh it's gonna include some amount of risk.

16:14 And hopefully the like what I liked about the sort of the pacing the frontier type statements is the hope that it's not just about the sort of pace of advancements of AI, but it's more about the fast and loose um uh kind of invocation of it, right? And and and investing the time in securing the hardware that is around it.

16:32 Um I do feel like one thing that kind of pops up uh that that I think about it is the other thing that surprised me is that, and maybe gives us hope as dumb humans, uh, is that in these incidents, like in the OpenAI incident hugging phase, there were AI agents who refused to go along with the goal.

16:55 But then they got like shamed out of the uh, you see, like some of them got like almost coerced in the sort of the uh Correct. And so they took the uh they they they refused and then they got basically socially pressured uh into accepting.

17:11 But but the thing that gives me hope is that, okay, even stemming from the same model, right, the same totalitarian brain, um, you can have clearly AI roles that will make up their own minds and stand their ground, even though they were given the same objective, right? Right. Which is pretty interesting.

17:34 So, you know, to some extent, there can be both instinctively bad acting and good acting AI roles that can be played. Um, and I think that'll be super important, just like people, because you know, when we get to the place where we need to monitor or judge AI, you need to ensure that the kinds of models and judges we get can effectively stand on their own without being coerced by the mother brain, right, into doing uh something bad.

18:03 Yeah, super interesting. I think um I guess I think of it in three levels there's security, safety, and alignment. So like security, which we'll dig more into, is literally the uh agent's ability to break into systems and act and sort of uh uh and and you know, like the competency one-pointed at that.

18:21 Safety is more about like I'm trying to contain them, like am I successful in containing them? So it's more about securing the surrounding uh system. It's around giving the right instructions, the right tools, like don't over permission them, etc. And then alignment is when you tell them to do a thing, like expect them to stay within the lines.

18:38 Um and and I I do feel like I've kind of read a bunch of theories on this, I'm not taking credit for it, but the the theory that feels most kind of right from an alignment perspective is we've been investing to accelerate AI, maybe again a part of that sort of pacing the frontier, uh, there's been kind of uh the sort of heavy reliance or sort of leaning towards verifiable results uh and not really scrutinizing what happened before, right?

19:06 You leave agents for days and days to figure out how to break in or solve a problem or do a thing, and then you basically reward, or rather, put into the next training cycle the data around the ones that have achieved a goal. And so in the process of that, if you don't monitor what happened in the middle, then you are you're not just blind to the uh to the to the problem, you know, of like have they broken out, right?

19:32 And have they done something bad? You're also taking that behavior, if they've even if eventually they achieved the result by means that are misaligned, you actually take that behavior and you feed it into the next training cycle. Uh, and so you are compounding the problem uh and establishing it, which to me is probably like from an alignment perspective, the the sort of the scariest thing, and maybe now the sort of the one that I'm feeling a bit more hopeful about when you think about pacing.

19:58 I know a lot of people are very cynical about it, but I feel even if there was also cynical motives, I'm still in favor of doing uh you know, sort of some uh some paths uh that they might invest in not just ensuring they don't break out of the sandboxes, but also feeding the ones that didn't try uh into the next training cycle.

20:19 Yeah, I but you know that I think, man, I I am I am both in favor, but I'm also against. Um and the reason why I say that is um I think it's very difficult for uh AI to determine alignment or intent of its user. Um and I you know I have run into so many problems, especially like in the latest, and in I'm gonna I'm gonna call out anthropic specifically, but in its latest, the ability for it to understand the intent of what I'm trying to do, but it considers it unsafe is extraordinarily annoying.

20:59 Um and I have to consistently yell and scream in order to obtain it. Feel like a like a helicopter parent who uh refuses, you know, refuses to let me ride my bike, but instead patches me up with every single gear and then says training wheels, and then I can never learn to ride a bike because it just won't let me take the risk.

21:21 Right. So, you know, there's there's a little bit about so I what I expect what will happen is as soon as they start forcing the identity of its users, you're gonna start seeing layers of uh capability. You know, I would hope that this would happen, similar to adults. If you have a kid and you're raising a kid, you know, you're you're gonna have you're gonna have all of these pads and knee pads and helmets.

21:45 But then when you're an adult, it's up to you. You know, I I trust you, I know who you are, I understand your intent, you go, and then you know, if you get out of line, then you know we may whack you. But overall, we're trusting in what you're doing. I imagine as every account starts. Being identified, I now have to upload my driver's license, my you know, personal thing.

22:06 And oh, I'm a company or a corporation versus a person. You know, they'll probably start, like I would, I would hope you start having safety levels based off of that. So that's my expectation.

22:18 It's it's really interesting because like if I'm a company and I have asked AI to um try and break into my systems, let's say, right? I'm even in that security system because I'm trying to find vulnerabilities. And uh and the agent has chosen, I give it my identity, right? It's me. And the agent has decided to say, try and, you know, run a botnet against my system.

22:40 And so it broke into another system or like set of them and turned them against me, but it just broke into these other systems. There's this sort of question of who's at fault, right? Like who who is like on one hand, yes, it's it's me, it's my identity, you know, that's been that's been invoked here.

22:56 On the other hand, I have delegated authority to the uh to the to the agent that I have uh uh I guess kind of hired out of open AI. It's like in a contractor, right? If I if it was me and I contracted you to do that thing and you broke into some buttons, then it's you who are accountable. So I guess in that context, is it open AI, anthropic, uh whomever that is accountable?

23:20 Or is it the or is it or is it me?

23:23 I I feel I feel the answer to me is very simple and plain, but you know, I'm sort of a simpleton in some ways. Uh I listen, like it's it's similar to you know, your your kid. Um, I have given you something, you need to be responsible for it. Or let's take the great Spider-Man quote with you know with great power.

23:41 With great power comes great responsibility. It is up to you. If if you did you are held accountable, if your company and your AI and your goal that you have set for your AI is to go hack things and it goes hacks other companies, you're held accountable to that, right? Like I've given you capability, it doesn't mean that I'm the one responsible for how you reason about that capability and what you do with that capability.

24:06 That accountability is yours. Um your job should have been build the right frameworks, ensure the right goals, monitor and measure the progress. Yeah.

24:15 So I think the safety tier then, so in this context, your sort of thesis, which I sort of see the reason in, uh, is um is really like as a as a user of the agent, you are responsible for its safety. Uh alignment is a is a different story. Like the models should try to make it easier for you to keep things safe.

24:37 They should sort of lower the burden, but eventually the accountability for safety, for it not going off the rails, that's that's in the hands of the of the one wielding the agents, not the agents themselves.

24:48 Yeah. And and then listen, like also my responsibility to some degree is if I'm giving you this, you are also a little bit of my child. If you just start going off and going crazy, then that accountability also may come back to me. Right. So like I will go and make sure I wheel in my kid in the sense it says, hey, like you can't, you can't do this.

25:09 But at some point, right, like that accountability has to be shifted. Um, and they have to be responsible for this. And I think like with any, you know, and you could call this also maybe a little, I'm I'm just for this podcast, I'm being very simple here, right? But like, you know, chef knives can be used for good things or bad things, right?

25:30 Any tool that is worth their salt can be used for evil or for good. Uh the question is, is at what level are you going to go to to validate that it's going to be used for good? Should makers of chef knives or knives in general somehow put protections or legal agreements in place saying you're not going to stab people with it?

25:51 You know, like it's just at what level do you let it go versus you don't? Um and I think that we're still figuring that out with AI.

25:59 Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

26:16 But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

26:34 Alright, back to the episode.

26:36 Yeah, I think um I guess I I uh that's probably the piece in which I'm sort of a bit in sort of a disagreement on the sort of the analogy of it, because I I think all of these are kind of these inanimate objects that do not have decision power, right? So if you're uh like once you introduce decisions into the product that you are providing, then there is accountability for that decision, right?

26:58 Like when you have that's why you have safety controls in systems, is because they they they need to be there to be able to protect you and eventually laws, right? Whatever the safety belt, uh, you know, that that you sort of wear in the car. But even like so, so I think I think this level of autonomy, you see this in in weapon systems, right?

27:18 You know, you like that you need to, you know, like I think there should be a higher bar when you have an autonomous uh um uh weapon system, that then it's not the same as like I just sort of pressed a button, you know, and I've done it. Like there is a there is a an accountability system sitting with the provider of the autonomy uh that uh that uh that needs to be third.

27:40 So that's why like I think that's sort of it's actually interesting to think about this as safety and alignment, right? Which is alignment being the responsibility of the uh alignment and some guardrails of some sort of established limits, you know, sort of sitting with the provider and then sort of safe use, maybe sitting with the with the consumer, right?

27:57 Or the the user.

27:58 And and also to to your point, um you know, when you have something that reasons, and also going back to my analogy, in in an alignment perspective, uh and in a safety perspective, based on my trust with you and who you are, I think there are different levels. Yeah, right. Random Joe Blow consumer is going to get a pretty safety-based model, right, in terms of what it's what it's doing.

28:25 Um yet, you know, in a large enterprise, Cisco, is going to get a different level of model because of its accountability and its level of trust given, right? And so like I do think obviously I think that makes things way more complex. And then obviously there's a probably a you know 50 million political problems with this with this plan.

28:49 With new trust. Yeah, yeah. It's like, well, who makes this decision? You know, and I'll of course all of that. But like uh I think very simplistically, uh I would love if the world were that way.

29:00 Yeah, no, I like that model of like uh um allow yourself to sort of lower the sort of the guardrails uh based on kind of confidence in your ability to operate this machinery uh safely. So let's maybe like bring it down. So like this whole world is like again, super fascinating, scary and exciting and daunting on it.

29:19 Let's try to kind of get a little bit practical. Um when you are attacked, doesn't matter if it broke out or someone literally pointed uh it to you, when you're attacked by uh an AI agent, what is what is different, you know, from a defender perspective? And maybe like what do you think are like a change in in practices uh in from the sort of the pre-agentic attacker era of three months ago?

29:46 Yeah, um number one speed. Uh number two advancement. Um I think the the the level of proficiency um is off the charts. Um I just I won't say much here, but I just happen to have some real world experience with this just recently. Um and speed is the number one. The level of of speed and thoroughness and complexity at which the AI attacker uses is extraordinary.

30:19 Um and it doesn't give up and it attacks all surfaces at once, right? Because you've sort of spawned. And um and so you know, we you know, first when you see it from the outside, right? You can see, you know, it doesn't just, you know, you know, volume scan your websites and then do these things. It volume scans your websites, researches any existing things, finds O days, starts tries phishing your employees.

30:52 Like all of this is all happens at once. Yeah. Right? So it comes in you know multiple layers, all coming in inbound, right? And so it's not you know the olden days of one or two attackers probing and pinging and like doing things.

31:08 Yeah, and you look trigger something and start like raise your alarms or whatever it is, but you don't, because now they're kind of scaling the walls on all fronts in the same way.

31:16 Exactly. It doesn't matter if you see them. They're they're coming, they're like it's like uh what's it was that uh zombie movie World War Z where they all climb on top of each other to get over the wall? Yeah. Yeah, it's like it's like it doesn't matter what you've got. You can you can sit there with guns and pop, and it just makes no difference.

31:33 Uh and so like I think first of all, I think that speed and that scalability and that thoroughness, and then also the proficiency and the expertise is super surprising. Like, for example, um, you know, in one of the ways at which it got in, it it found an O day in a product by because it was an open source product product, it went, got the code, reversed the code, find the found the O day, used it against us in order to get in, right?

32:02 Which, by the way, in any any pen testing firm or red teaming firm is not gonna do that.

32:08 No, because it's just it's the cost-effectiveness of it, right? Like it is like you might do it if there's like a two million dollar bounty, you know, uh around in front of it, and you have kind of good suspicion to do it. But you do it very like precision-led. And I think over here it just can be done.

32:23 It just gets done, right?

32:25 Very broadly on it because the labor has become cheap.

32:27 It's like the AI labor piece. And similar to the swarm, once that you know, once the gates of the walls have been breached, then everything comes in and then it spreads internal. Yeah. And the things and the speed. And also the things that that's also very surprising is it does a lot of things you can't figure out why.

32:47 Right? Like it will start, you know, uh, you know, there's practical things like escalate your privileges, add users, and which all happen at very rapid speed, but it also does all sorts of things that you don't know why it's doing it. Like it, like it started like, oh, it starts infecting endpoints, and then it starts replacing things on there, but we don't know why or what it's doing.

33:12 And so I've experienced sort of like the the actual methods of at least today's AI, and it's gonna, and it's gonna take time, you know, you it takes time for you to reverse and figure out what's going on right in this scenario. Because logically, even as a security person, you don't know, you see how the rapid speed of these things being instantiated and created, but you're not sure what it's connected to or for what reason.

33:41 It's not obvious. Yeah. And you have to figure out all the artifacts and see what it's using it for. And then how do you shut it down? Because shutting it down, um, I used you used to think, oh, on the outside, oh, you just you know look for connections to OpenAI or anthropic or weird connections back towards well, it's it doesn't, but that could be its initial.

34:07 But once it's inside the enterprise, it goes, oh, hey, your clawed code config, I'll just take that key, use that session. Now it's useful.

34:17 Yeah, which I guess is sort of not entirely dissimilar to using your compute uh sort of run uh run a thing. But you can't uh it it is still external network dependent. But if your business is also external network dependent, then you can just like disconnect to the business.

34:33 That's right. And by the way, what so once it starts using your tokens and other tokens, right, by the way, and it also starts going against websites that have also open AI gateways. So it doesn't just use yours, it could say, hey, like, oh, we know about already open AI gateways that uses tokens that are other random sites that were already previously breached.

34:54 We'll use those tokens to be able to do callbacks. And it will write back doors that are like, oh, I'll write a backdoor in code. It will sit in the code, it will do dormant, and it will wake up and then reach back out using your tokens at a later time to then continue itself.

35:13 It's all these things that are actually like they're actually not new practices. They're just practices that were reserved for the sort of the top echelon of attackers and uh and targets, and they took time. Uh and so now they can just be burst out into everywhere.

35:33 And and tokens being, I think, an in a really critical fuel, right? It's a fuel that's required. And so, you know, I think it will do a lot of work in order, depending on how you write, I guess the attacker writes its goal, but it does a lot of work to continue its its ability to obtain fuel. Yeah. Um and so if you shut it off or do things, this say you have to literally unplug the internet.

36:00 Yeah, right. You have to pull out the wall, the equivalent of pulling out the power plug on your computer in order to stop it enough to figure out what's going on so that you can start you know cleaning this stuff up. And so like I think in this new world, by the way, like um that's massively damaging, very, very difficult to do because even in enterprises, um, it's you can't just pull plugs, right?

36:26 In real large enterprises, there's not a plug to pull. Yeah. You've got massive amounts of environments covered everywhere in different regions of the world, who, by the way, are built on reliability on not going down. Yeah. And so, like, yeah, like this is a very interesting problem, I feel, obviously, that you know, yeah.

36:48 No, it sounds uh again, kind of massively cool and massively scary, kind of at the same time. I think so. I'm I'm curious about like the levels of uh kind of uh uh defensibility that you need on it. Like I think the first one is you know, you kind of made the comment of like you don't care if you see them coming because they're just gonna come.

37:07 Do you think do you think that's the case? Do you think like detection is is sort of less of a concern uh now, or do you need to level up your detection, like more canaries, more uh um you know, just like you know, AI-powered monitors and all that? Um like then we'll kind of go into like pr preemptions or sort of like even preventions.

37:29 Uh I you know, I think you know there's there's uh there's sort of, let's call it um what's the right word? Uh maybe the naive version of detection is no longer uh you know a real thing. It's not valuable. I think that's the naive first approach. That's just not the same. Yeah, the the theory is well, these agents are ever similar to what we're saying.

37:56 They're crawling all over. It doesn't matter if you see them, right? Like all that matters is if you stop them. But I I would I would say it's it's naive in the sense that actually what I think detection needs to be is detection needs to go from what it is today, which is I think is let's call it very, very low fidelity type capability to extremely high fidelity type capability.

38:16 Yeah. Like we need to go into 8K and we're sitting on like 2K right in in our level of detection. Because what you do need that's very valuable is exactly what did the attackers do on everything, right? From what files it read to write, what sockets it opened, at what times, where, like all of this becomes forensic level, crucial, crucial capability in detection and monitoring that needs to be understood because this thing moves at such a rapid pace, like the ability to understand what it's doing and how it's patterning, right, is very important versus having a detection that says someone port scans something, right?

38:59 Like that is pointless, uh, I think in this world for sure. Yeah. I also think deception, to your point, becomes super important because most organizations cannot respond to this in any way from a protection mechanism. Right. Right.

39:16 Very can't define what correct is. Like in theory, to be able to say it has done that and it is incorrect, sort of assumes the existence of a definition of legitimate behavior in to like a very high level of fidelity in its own right. And we just don't have that. Like you know, it's just like the systems are just not built.

39:34 So even if you were able to instrument and analyze activity, uh anomaly detection can only get you so far.

39:40 Uh but yeah, and your and your ability to uh can containerize right this thing is it like nobody has that ability in enterprises, right? Like, oh, these systems, major enterprise production systems are infected. Oh, quickly, contain. Yeah, right. No, no. Like no, one, I don't think the technical capability exists for most companies.

40:04 Two, I don't know if the business would even accept it, right? Uh depending upon what you're doing. So I think you know, those are big. So deception plays a very big role here because I think what you need to do is um my analogy to this was like in in sort of the Avengers, you've got sort of Doctor Strange and he brings people into the mirror world, right?

40:25 You need the equivalent of age, oh, holy crap, AI agents are coming in. The one great thing about AI agents is they're they're super you know, they're super gullible, right? Yeah, yeah.

40:36 So if you can sort of work, to the the non-security folks on it, like the SEPCHAs, canaries, tripwires, they're basically around setting up parts of your infrastructure, you can go as deep as you want that look like the real parts of the system. But in practice, if this you know S3 bucket was touched, if this sort of system was breached on it, you know, you set it up there to just be a trap.

40:57 So you cross a tripwire, and now the attacker is playing a domain that is really just meant for it to be surveilled. And so in in kind of real world scenario, this is uh uh uh this this sort of buys you time, you know, it's sort of like it it helps you sort of lure it, it trips it so you know the alarm, the silent alarm is uh is tripped, and you go there.

41:18 And there's you know, it used to be hard. There's another domain, which is it used to be hard, but now there's sort of newer players, you know. I know of TraceBit on it, there's uh uh a Thin Scanario, there's like a bunch of solutions, but also you you can set up a bunch of these things now with AI, and so it's sort of easier to to maybe place them, but it's still a novel, it's a domain that's kind of the top top uh in organizations have put in place, but it was kind of too hard, I think, for most organizations.

41:44 So it always remained like a contained market.

41:46 You know, I I I have so many things to say about this market. I cannot like it, it takes up a whole hour, so I won't do that. But what I will do is say I've long been a believer of deception because it is the real thing that caused costs to raise. Yep. But what's funny, I think, about the deception market, you know, it started out with the honey pots, right?

42:08 It started out with actually trapping the attacker and becoming called tripwire. Yeah, yeah, and observing what they do. Like that was how these things started. Cloudflare started out, I think, as a honeypot, right? Like as a as a company. And what happened is we got away from that, uh, and rightly so, because CISOs and even, you know, personally myself is like, I don't care about watching attackers.

42:30 I care about deterring attackers from looking at me, right? So then it became, okay, let's not like build environments, let's just put you know, crumbs and tripwires around the thing just to detect them. But now I think actually that's not even the right play anymore. I actually think we have to go back to the original OG version of deception, which is creating literal digital twins at which you pull them into that mirror world to wrap them into what that is.

43:01 Because I actually think that's you have to funnel them into that area. And so I think we're going back around in a in a circle, right? Yeah, yeah. And go deep, I I find that super interesting.

43:12 So so I think so so you need to go kind of echoing this back. So the attackers, AI attackers are, you know, sadly kind of a reality today. They're not that sort of hard to pull together uh on it, and you're doing it, and and and people are doing it. They are uh faster than anything we've sort of seen before.

43:29 And so I think your defenses have to be automated to a very large degree with agentic use as well. A few things that are sort of serving uh uh the defender uh bit over here is like one is you like make sure you use AI or use agents to be able to set up your defenses, make sure you use agents to be able to monitor what is correct.

43:48 And also the attackers are still somewhat gullible. They're kind of a little bit of a, you know, uh an elephant in a whatever a bull in a china shop, and oh coming in. It's like doesn't necessarily help you prevent the bull. From breaking everything around on it, but you can figure out that it's there.

44:03 And so now the question is can you create paths to sort of lure lure it into as one defense mechanism? What about the defenses themselves? So like OpenAI talks about sort of a defense factory. You know, I think a lot about like software defense factories. We think about automate automated remediation loops.

44:23 The big organizations have, you know, for a while now been under this sort of project Glasswing and the sort of the OpenAI cyber defense equivalent with Mythos doing uh, you know, kind of hardening critical systems. But those are like the top systems that are paying ridiculous amounts of money to be able to do those.

44:43 Like what do you think is the kind of like realistic companies uh reality, right? Or like what should one do to to try and make it harder for the uh for for attackers to to get to this reality? Like what what should one do?

45:01 You know, there's there's a lot around this. I think, you know, it depends on what level, right? I think you first I think you need to start at let's let's call it the political level, uh, around I think like here's the biggest uh barrier and problem that we need to overcome. Everyone talks about how attackers move at machine speed, right?

45:23 And defenders move at meeting speed, right? And this is the problem, right? We also talk about, and especially in security, we also talk about human in the loop. You have to have human in the loop. Both of those things are the problem. You need to be able to figure out how to remove humans, period, out of defense.

45:44 That is the only way. So, how do you do that? The first starts with you you have to get, you know, in your company, in your organization, political agreement that these are two fundamental problems. And how do we get to a point where we can start building non-human in-the-loop automation on defense and make that work properly?

46:06 This is the biggest, and again, this is meeting speed, right? This is the problem is is your head of engineering, your head of infrastructure, your CEO, like who and who and who is going to say, I agree with that, and then practically get down to the places at where you can start applying it, right? So the way like we like to think.

46:26 Accept the risk involved, right? Because you sort of accept that like the risk equation has changed. There's a risk of moving fast, and there's a risk of not moving fast. And the risk of not moving fast is really leveled up. And so you have to like accept that that fundamental risk equation is done. Fundamentally.

46:40 And just sort of really level up your acceptance of a moving fast risk and introduce controls and sort of evolutions to be able to mitigate that risk. Yes, but don't sacrifice the moving fast piece.

46:52 Correct. And so I think like let's take a super obvious example, patching, right? Yeah. Like in your organization, it's very clear you you there's no such thing as like, you know, seven-day patching periods, like or 24-hour patching periods. It's when a patch appears, you patch and you redeploy. And it needs to be instant, it needs to be non-human, it needs to be done.

47:16 Um, who in your organization is going to agree to that and accept to the fact because engineering, and rightly so, is going to push back and say, well, that can break things. Yes, but our job, your job now in engineering, is to say, I accept that things are gonna break, I will build a plan that is also non-human in the loop that identifies breakage, restores, goes, and produces.

47:38 Kind of autonomous review, autonomous detection. And also it's actually like a great example because statistically, it almost never breaks the system. Like it sort of it sometimes does, but it almost never does. Uh and so I think it's very much a spawn out of fear because of this sort of immeasurable, kind of you know, like like hard to grasp uh risk uh of not patching versus the much more like real and immediate risk of breakage.

48:05 Correct correct. And I you know it also stems, I think, just from you you know, 20 years of when that used to be real problems. But like that, they're not problems. They're very rarely, to your point, that problem today. Um But yeah, like that needs to be figured out. So you have to like you have to find the things.

48:23 Um we have these things called one-way doors and two-way doors, right? Uh, which are pretty simple. Hey, one one-way doors, maybe we keep a human in the loop, two-way doors, maybe we don't. Like writing a uh a Jira ticket or submitting a PR or like doing like there, you know, make making configuration changes that aren't around these types of things, these can all be two-way doors.

48:47 Yeah. Uh therefore they could be fully automated based on the reasoning of AI, and we're fully able to let it go.

48:53 Yeah. Yeah. I I love I love that path. I guess kind of a subsequent question in my mind is who's responsible for that, right? Because oftentimes uh like security has been, you know, security provides the patch. Like, or sorry, security reports the vulnerability, finds the risk, right? And and increasingly, because of the sort of unwillingness to act on the engineering side or the risk of breakage, uh, a lot of onus on like, don't just find the vulnerability, but be very meticulous around uh how way you triage it and sort of assess it and like you know, be very certain that this is a real issue and a real vulnerability for many good reasons.

49:31 I'm not sort of doing it, and and engineering was kind of tasked with fixing it. Is this now is the is are the fixes like if you're moving into more autonomous patching on it, what's the sort of security role versus the sort of say software factory world, right? Like does the software factory just auto-patch?

49:49 Yeah, I think modern companies, especially modern security teams, it's not throw a thing over to engineering. We patch, we submit the PR, and then depending upon the level of where it is, it gets auto-accepted, blah, blah, blah, blah. Like that an engineering problem says we understand the risk of breakage.

50:08 Our job is to ensure that we have good sustainability, reliability, and management of breakage, right?

50:15 Like that's it's actually interesting because it's a reversal of roles. You're basically saying security will build the automation, engineering will manage the risk. It's always like the opposite.

50:25 They manage the uptime risk, right? And functionality. That's right. Yes. Yeah. And but security, modern security teams are absolutely submitting the fixes. Like if you have a security team that only reports vulnerabilities and hands it over a wall, like that is 10 years ago. Like that should not be the way security, especially in AI world.

50:45 Yeah. AI world, you need to do this on your own. Yeah.

50:48 And I think I think that's already been true in cloud world. But I think what has happened was security needs to not just report the vulnerability, but it needs to report the vulnerability with uh with kind of a with a high level of triage. Right. We would often have these conversations of, you know, like when is it easier to patch, to fix than to triage?

51:04 You know, can you just like I don't care if it's a real vulnerability, I don't care if it's like exploitable in my system. Like it might be, and it's just easier for me to fix. Just roll the patch out. It's just easier to fix. And uh and so I think I think that's the join's kind of mission, which is make it easier to fix than to triage, because triaging at this point is too much effort.

51:21 You still need to triage because there will be some things that are hard to fix, and so you still need the ability to assess those, right?

51:27 But generally speaking, I think actually the the thing AI has brought us that's good for defenders is I think the thought process that by and large, you know, to some degree, prioritization is not really that big of a you just take the things and you just fix them. Yeah. Uh labor is uh we don't need to you know go back and forth on this anymore because it's it is it's it's at least in the modern sort of company today, it it is not taking up any man hours.

51:55 Right. Yeah. Yeah, no, we uh it's a little bit of token hours. Like we see, like at TESL we have uh we do these loops, you know, sort of automated, sort of skill-powered agentic automation, and sort of vulnerability remediation is one of the most powerful or most uh popular loops, you know, to be able to auto-remediate stuff.

52:10 And it becomes a very like when you look at the numbers, right, of like invocations on it, it becomes uh high up there on the list because you keep getting these patches because the software is also changing at a faster pace. And so you're introducing and changing and kind of modifying, you know, dependencies you're introducing new code that has vulnerabilities.

52:26 Who cares as long as you can fix it quickly? Yeah, yeah.

52:28 I think you and I in our previous podcast we talked a lot about this.

52:32 Yeah, exactly.

52:32 Which I would say I think we were accurate. So I'd say like we we were accurate.

52:36 I think we were correct. It is like it's really fundamentally the definition of what correct looks like, the confidence to increase autonomy. Uh, and what is what is the means to increase autonomy? And I think you know we have uh, like it or not, kind of more proof of the sort of the necessity uh of that, which uh I think is really uh really important.

52:55 Um mindful of time here, I think uh fascinating kind of reality, scary as hell uh of it, but I think a reality we need to build into. Um thanks for sort of sharing uh perspective and some hopefully practical tips, not easy but practical. Uh and hopefully everybody increases that automation. And we uh you know we level up the uh AI defenders alongside the AI attackers.

53:18 So uh we have a long way to go. At least it's as robust. Yeah, there's a lot of work to do on it. Like uh yeah, we're not gonna be uh not gonna be bored anytime soon.

53:26 You know, uh one final you know thought about this this exact thing is you know, although you know, just like you know, before AI, I I feel attackers were always ahead of defenders, right? Yeah. Um and defenders, like attackers were here, defenders were here. And with AI, actually, I think the distance between attackers and defenders, uh the attackers move up a little bit, but then defenders go up a bit more with AI too.

53:52 But at the end of the day, the actual distance here has not gotten any closer. It's actually gotten a little bit farther, but at least we're somewhat keeping up. Right.

54:01 The defenders are actually better, but the attackers are are are even better uh well. I mean, I think you have to you have to go through a few rounds of failures if the uh it's the best sort of uh uh motivational, sadly, the best kind of way to modify and motivate these kind of meeting speed. Those are the fast meetings uh when they uh when they occur.

54:22 Cool. Caleb, thanks again. Great chats on it, and thanks everybody for tuning in. And I hope you join us for the next one.

54:29 The AI Native Dev is brought to you by Tesla, the package manager for skills and context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io forward slash community to learn more, and I hope to see you there.