Your Coding Agent Should Never Hold a Credential

0:00 There's a very famous incident called Pocket OS, where unfortunately, you know, the user of the agent wanted the agent to do something in staging, it found keys in production, and the production database caused a significant amount of damage. This happens all the time. You know, agents are more and more happy to escape sandboxes, right?

0:16 And while in most scenarios, and the wide majority of scenarios, an agent will do exactly what you ask it to do, in some percentage of use cases, it will do things that are not your intended action.

0:34 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Fajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

0:57 Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works.

1:21 This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. As a podcast listener, you can get 15% off your ticket using promo code POD 15.

1:45 We have so much planned, and we can't wait to have you join us in New York. Hello and welcome to another episode of the AI Native Dev. This time we are recording this live from San Francisco at the AI Security Summit run by Sneak and also the AI Engineering Group. Here we have a wonderful panel, and we're going to be talking all about agent baseline.

2:12 So this is going to be a fun session. Who do we have with me? So on my direct left, uh we have uh uh Ellie Aylener uh from Docker. Uh tell us a little bit about what you do. Uh I work for Docker.

2:23 Uh I'm a VP of product strategy Docker. I help uh uh Docker to work with some of our most strategic uh customers and help kind of direct some of the initiatives we do.

2:33 Amazing. Uh next uh Ellie is uh In Livingston, uh who actually was on the podcast not so long ago with Guypo. Uh you work for KeyCard. Tell us a little bit about yourself.

2:42 CEO of KeyCard. I've been working on agent identity and access. So, how do I adopt agents? How do I build agents? How do I control what those agents can do across the board? How do I make my developers and my business users successful as possible with agents today and into tomorrow?

2:55 Awesome. And finally, but by no means least, uh, we have Marcelo Sosa from Sneak. Tell us a little bit about yourself, Marcelo.

3:02 So I basically help customers uh safely adopt AI, build agents, uh, you know, bring uh you know defensive capabilities into the hands of the defenders and uh just try to protect the world, you know, bit by bit. Awesome.

3:15 Now we're gonna be talking about agent baseline. Ian, why don't you kick us off? Um in a sentence, what is agent baseline?

3:21 It is six controls. Uh, and those six controls are ways for you to think about what security controls are need in place in order for me to take advantage and use autonomous agents in my organization or agents at all of my organization.

3:33 Do we really need secure agents? No, you didn't need to answer that. Uh Martello, who is this who is this for? Is this for a CISO or is this for engineering teams that are actually building with agents today?

3:43 I mean, the way I see it is that you know it's really for people that are trying to like understand like what are the security boundaries that they want to have when they're basically adopting agents and getting agents to do you know autonomous tasks. Uh obviously we are focusing a lot on the software factory.

3:56 So like, you know, people that are actually using coding agents, and so like you know, why there's a security persona, obviously it also matches up on the platform teams, you know, like how what how they use like dev containers, for example, and kind of how they get the sort of boundaries of where agents uh uh are operating.

4:10 So we kind of see these multiple personas, uh both on the security and the platform teams, like playing a role here.

4:17 And you mentioned software factories there. Um Ellie, what what would you say is the definition of a software factory in given the context of this uh this paper, this this meeting?

4:26 I think uh many people define software factories in various different ways. Uh fundamentally, it's what organizations use to produce code. And there's various various levels of autonomy uh that exist in software factories. So there's anything from the first level, which is giving developers uh coding agents, right?

4:47 And how do you secure that? How do you allow developers to be able to produce code where they're the ones directing uh the whole process end-to-end? The second level of autonomy, which is uh you are at that point having a bit of a workflow, and in that workflow, you're using agents uh to produce code, to test code, to do various parts of your implementation.

5:10 And then uh, but the second part is that the workflow is predefined. And the third stage of software factories is where the actual there is no predefined workflow, it's actually the agent who is making the decision on the next step to take, the right things to do, the right agents to call, and so on and so forth for that.

5:29 So, and we're seeing companies be in various stages of that maturity, and actually what we're seeing is we're seeing that the depending on the task, it might make sense to be on the on a various levels of that maturity curve. We're yet to see many organizations that have made it all the way to the third one and have done that successfully, but we're seeing companies be on that journey, and what we wanted to do is we wanted to put a paper together, kind of applying the principles that Ian talked about of agent baseline, on how to think about how to build software factories utilizing the principles we outlined in the paper.

6:06 Yeah, and so for people who I mean this was released this week, right? Uh in fact Tuesday. So, yeah, it is it is, I say this week, like like this is being live streamed. At the time of recording, it is the 15th of October on the on the on the Thursday. This was released on the on Wednesday, the 14th.

6:21 Um now, in this, there are like you just mentioned, there are many levels. There are three levels. There's also you know six kind of pillars of of what people should be thinking about. We're gonna go a little bit deeper into that. But for folks who want to uh read all about this, uh agentbaseline.org, and you can take a look at the the V1 of the reference architecture.

6:38 And I think this is open for comment now. So please take a read of that, have a look at that and and give your feedback um to that. The team are very interested in that. But let's jump into this a little bit into a little bit more depth and I'll and I'll maybe pick on a few a few people here based on your experience.

6:53 Um so Ellie, um sandbox network boundaries. Um talk to me as if I've never set one of these up. Why is it important and and and explain to me the uh what I need to do?

7:03 Uh I uh uh uh hugging face? I'm hearing I've I read I read something about that. So yeah, so I mean don't further. Uh maybe take a step back and think about what we are actually doing over here. So if you think about agents, uh on one hand, they're just software. Yeah, on the other hand, they're directed by undeterministic uh uh actor or undeterministic thing that they're trying to use, which is fundamentally probabilistic in what it's trying to do.

7:36 And while in most scenarios and the wide majority of scenarios, uh an agent will do exactly what you ask it to do, in some percentage of use cases where the percentage is not zero, it will do things that are not your intended action. And it might be in most in most situations, it would be in good faith.

8:00 It will try and do it because it's trying to achieve a task. It's it's it's really good at pushing through and trying to achieve a task, even if it means potentially, you know, doing some malicious actions. So, what you're trying to do, you're basically trying to take an actor that could potentially be malicious and could do it at very high speed and do it in swarms, and you're trying to give predictable uh things that this actor can and cannot do.

8:31 So, what you're trying to do is you're trying to manufacture trust of an actor that is fundamentally unpredictable. So the way we think through this is what's needed is is needed, what you need is you need a runtime that can enforce boundaries. And the boundaries are you can't get to any network you want to get to, you can't get to secrets you shouldn't be getting access to, you can't get to uh uh folders or I don't know, keys that are stored in your local machine in order for you to be able to trust that particular agent.

9:03 So, in our mind, in order to manufacture trust, in order for you to be able to have autonomy for those agents, you need to put them in environments where uh you they can only do a certain set of tasks, which are predetermined ahead of time.

9:22 And in order to get that autonomy, um it it potentially potentially will need certain access to things like keys and credentials, but but Ian, why should an agent never hold a credential? And what is the what is the way we can allow autonomy for agents without them having that access?

9:42 Yeah, I mean I think fundamentally, if you uh using Cloud Code, Pi, you know, harness on your computer today, uh running inside your user space on the operating system, and you give it your your secret. That secret, whether it's like uh let's say it's a Stripe API key or keys to AWS, there's nothing that prevents that agent from performing any action those keys enabled it to do, but that may not be actually your intention of what you want the agent to do.

10:06 And so there's a very famous incident called Pocket OS, um, where unfortunately, you know, the user of the agent wanted the agent to do something in staging, it found keys in production, ended up leading a production database caused a significant amount of damage. This happens all the time. There's a lot of stuff going on in personal agents today where the agents are doing things, the personal agent's instinct, muse, and friends are going on doing things the user didn't necessarily intend.

10:28 So at the end of the day, it's a question of like what's the boundary and how do I prevent it from doing things. And traditionally in today and into the future, it's always what permissions does this agent have in the context of what it's attempting to do, and what how is it passed that around? I think the other component of it is you don't want the agent to be able to find keys it shouldn't have access to, but also you don't want it to accidentally leak keys it has access to.

10:49 So an agent's not particularly smart, and it could just write a key into disk, or it could send a key off by Google query parameter or put it on the open web without realizing it. And so from that perspective, you have to, you know, put in a sandbox, make sure there's no secrets inside the sandbox. Uh and then most importantly, you want to have that on the egress point as traffic flows out.

11:08 That's where you really want to have your enforcement and also do add any type of secrets that are required for oldbound traffic. That way you don't have this issue where the agent's leaking keys or using keys, you can also give it a.

11:19 Exactly.

11:20 And then you also have the the beauty of being out of an authorization boundary. So many keys today, of course, give you more access to do things than they should. And so you can also implement authorization boundaries to scope those keys down to exactly what they need to do at any point in time.

11:33 So it's almost like a child, it's almost like you're treating a child, right? You're giving it boundaries, uh, whether that's whether that's you know, giving it containment or or you're you're stopping it from getting access to things it shouldn't do. Um Marcelle, let's let's talk about supply chain.

11:46 This is another this is another key piece of this of this of this reference architecture. What does supply chain mean when the thing picking these dependencies is an agent? What is it how how does that relate?

11:56 You know, the way I see it is what is the sort of you have to you have to think what you're trying to protect, right? It's like so like when you have, for example, you put an agent on some sort of sandbox environment, like uh the risk of the agent installing a malicious package there is bad, but it's very different than you know, like you don't have any sandbox uh whatsoever, right?

12:14 You're basically installing a malicious package on like you know a developer machine, for example, which is a formidable target today. So for me, when I think about supply chain, I think about okay, what is this coding agent actually pulling in and what it's actually using as a way to like uh communicate, right?

12:26 So like you know, when you think about MCP servers, when you think about uh skills, like what are the skills that you know is actually loading. And what a lot of people don't realize these days is that you know, like when you set up these coding agents to do a task, and one of the tasks may involve like cloning a uh third-party repo, like an open source repo, and then suddenly that open source repo you know has configurations of agents themselves, right?

12:47 You know, like you know, skills and and uh and and mcp server connectors, maybe the agent is gonna start you know actually consuming those as well. So like you have this sort of supply chain that is like almost dynamic that is happening. It's not like you can just look at your package.json and see what package is actually like being being pulled from that particular application.

13:02 So we think of it as like different types of risk that you're trying to contain uh when when you have that. And um one of the very interesting things that happen uh is this shy wooled uh warm, right? Where basically it was a very specific attack that at some point actually just spawns a local MCP server and try to actually do exfiltration of like you know sensitive data based on that.

13:24 So like we actually always should be treating things like you know untrusted MCP servers that the agent is connecting to as like you know a potential like you know severe uh threat factor to to kind of really uh mitigate uh immediately. So those are some of the things we think about when you think about uh supply chain of of these agents running.

13:42 Maybe uh just to continue your analogy of kids, you know, uh one way to think through this is like if you have a kid, you have to be worried about the things that it consumes. Yeah, right? So supply chain, it can consumable like the fact that that that you've put an agent inside of a sandbox does not mean that it can't consume things that would make it do things that are pretty that that are very bad.

14:07 You have to you have to worry about the supply chain, you have to worry about the things that it can do on the on the on uh when it does actions. But do assuming that just sandboxing is enough, uh, in my mind is completely insufficient. Yeah.

14:22 We we know that you know agents are very more and more happy to escape sandboxes, right? Like, you know, as soon as you give it some particular access to a machine which is insecure themselves, right, they basically just port themselves into that and uh and actually start doing damage from that instead of like the sandbox environment.

14:38 So let's continue this analogy with uh treating agents as kids.

14:43 So you remember someone has to orchestrate the children.

14:48 This is where we're going. Yeah. I often refer to myself as a babysitter when I should actually say parent. So but yeah, orchestrator I think is really my new term. Um okay, so Ian, the the the the paper the paper admits that sometimes it you know we still don't know how to safely work out an agent's task scope from untrusted input.

15:06 And I think this is uh I I love referring to this almost like as control plane and data plane and how they actually how they actually mix. And I guess for the kid, this is um you know whether they take instruction from a from a responsible person or maybe an irresponsible person and how they actually how they actually determine the difference.

15:22 So given that we the even the paper admits and we still admit it it's very hard to be able to distinguish between that, what is a what is a reasonable answer of actually how we can we can help ourselves in in determining what's what you know what's a responsible way to deal with this potentially malicious input or or untrusted data?

15:39 Yeah, I think I think the the admission in the paper, and I think this is true to synagentic security, is we have to accept the fact you can't trust the agent, you can't trust the inputs, and you want to control um, you do want to control the inputs can come from, but it's an almost an unsolvable problem.

15:53 Some of the inputs can be controlled. For example, you know, the base image inside the sandbox. Like what's the environment inside the sandbox? Are there potential supply chain vectors coming in? Well, something that's an example that's very difficult to control is like uh prompt ejection. Yep, right?

16:07 It's I don't know if we'll ever be able to actually solve that problem specifically. And so instead, what you have to say is you want to basically say, we're not gonna have it be wide open to everything. Uh, we're going to create hardened boundaries on everything we can, and where we can't, we're gonna try and pursue the principle of least privilege and have, to the best of our ability, have um over time, we're gonna have agents that are actually monitoring other agents, and those agents are monitoring other agents are gonna be very responsible for trying to ensuring aligning that the current agent's aligned to the task and it doesn't get the same context, the same openness to prompt injection

16:40 that this one does. And that's how you kind of keep those two systems in alignment. And I think broadly speaking, the best way to think about it is nothing about this is trustable. So the only thing you can focus your trust on is how do I create as much of a hardened boundary around the untrusted thing so that I can harness the value of an agent, which is really this do while loop that's just gonna experiment things on a hypothesis over and over again.

16:59 Want that to enable it and get as much of a sandbox or playground as you will, for it to try building a sand castle and iterate its way to success in a non-destructive fashion.

17:09 A couple of interesting things then. So least privilege, this is this is something that comes up again and again and again when we talk about AI. And it's just like it's a it's a it's just a good bra, it's a really good practice that we should be doing anyway, but we shouldn't stop doing it because of agents.

17:20 And the other thing that I really like you saying, so that that that stops things from blatantly doing things it shouldn't do. And the second piece that you kind of mentioned is almost like having an agent as a judge to make sure it's it's it's likely doing the right thing, not just doing something completely uh ridiculous.

17:36 Well, and I think one of the important things that actually come out in the last four weeks or so that I think is important, and I think we'll define the next 12 months of like of agents and what we're going through is decision models. Right? I think those those actually give us a capacity to have something that's far cheaper than a language model and is better atuned to making you know decisions on whether something is aligned with intent or not.

17:58 Yeah, with confidence levels as well, which is super interesting, super important. Um Ellie, if um one of the things here that we that we talked about is is is high impact actions that that need human approval. This is always a this is always a fun um boundary. How do you how do you stop that um, I guess, for things that need human approval?

18:17 How do you stop that from becoming a button that people just end up blindly clicking without without actually reasoning about it or looking at it in that much depth?

18:25 So I think like what we've all seen uh I mean manifest itself from uh early days of cloud code. Uh like we're talking in this is this is October of 26. Early days of cloud code was maybe like a year ago. Uh but in AI years, this is this is a long time ago. Uh the the uh approval fatigue one would have from like the uh I I call it there is two modes that agents operated in up until recently.

18:59 I call them the YOLO mode, uh like you just let it rip, or the CYA mode, which is the mode like you're basically approving every single action. So like someone could eventually blame you for saying yes to that. But once once you are, you know, like the the reality is like as soon as you start asking people to the yes, yes, yes, yes, yes part of this, uh you I mean, are are you know it's it's been long known that people don't usually read any warnings you throw their way.

19:33 It's even better if you throw it to them every like minute. The chances of them reading any of that stuff is almost non-existent. So like the fatigue of approving every single thing is becoming it is very real, and this is why people are uh looking for ways to run things with autonomy and not approving every and especially and the models have become much better, yeah.

19:55 Right, where they can do tasks that are much longer, but you still want to maintain control and you want to be to know that this particular model will not do things that will cost you your job, uh, you know, uh, or other very adversarial things that could happen if an agent escapes or an agent is allowed to do things that it's not supposed to be doing.

20:20 Um let's talk a little bit about skills. Um now, how much should I guess security teams care about skills and you know, poison skills? Is that a supply chain problem?

20:33 I mean, I think if you take a step back and you think about why skills are important, it's to basically do continuous learning of the agent, right? It's just like you know, you can't you we haven't really solved like you know, in terms of like how you do model training of like how do you actually get like continuously get the model to become better and better.

20:49 So like skills are this sort of like matrix-like moment where you know just say I want to fly a helicopter, just put a skill, and and you get it get to fly a helicopter, right, as as a human. So I think if you go back to understanding what that means, is it's like you know, you can actually teach the agent to do something very well or not so well, right?

21:08 So like if you actually have someone tempering your skills, you can actually add it to like download certain binaries and execute them as they think they are actually doing that as a prerequisite for you know doing a particular action. So we have to be really careful about what are the skills, what is the skill actually telling or trying to induce the agent to actually start taking terms of action.

21:26 So I think of them as sort of like this learning or or context that you're adding into the into the agent, and then it's really up to you to kind of understand, okay, is this really something that you actually want to have the agent doing? And if you have something where it's very tight with like the alignment of the skill of like you know, actually going to follow, and because you can also run scripts uh with some of these skills, you know, as you're kind of loading them, uh that is probably kind of you know like a real supply chain problem there, right?

21:51 You're actually running uh people's code in their you know environments without them even wanting, or without the coding agent doing anything. So there's all these particularities about you know the way.

22:00 you all the skills the way you actually like get them to to like change the the way the the agent is is is is operating that actually should should make it a concern for for you yeah cool i i've got two more questions um i'm gonna i'm gonna end i'm I'm gonna be very kind and and tell you all about my last question I'm gonna ask you all um what for people who read read this entire uh uh reference material and say yeah I totally agree with this what should they when they come to work on a Monday morning what's the one thing that they should do straight away so that's my that's gonna be my ending question so you can think about that each and I'll ask each one of you that but what I want to know is

22:37 we have sneak here we have uh uh keycard we have Docker and actually me from Tesla who who also reviewed the dock um I'd love to know what is the you know you're you're all you're all different companies different points of view what's the one thing that you all disagreed on what's the hard what was the hardest thing to find parity around what do you think what what was it Ellie I mean like uh it's easier to answer what we all agreed on but that's why I asked the other one I got it we disagree on the least I think the thing that like you know I think when in writing the paper the hardest component and it took me like actually about a large amount of time after everyone stopped talking and I was

23:18 kind of left with the pen to sit back is like how do we take this concept of the software factory which is like one nebulous at best right who who knows what that is and then two how do we break that down into sort of the different tiers and levels of sophistication so that you know the next question you ask is how do I map to where I am today?

23:38 Because I think that the the truth is is there's so much I won't use the word misinformation but there's certainly a lot of like marketing going on around around these concepts like from vendors and from people who are doing it and saying hey I'm merging 90% of my code automatically and it's really hard I think for a lot of people to say well I'm really like we're just like we yeah we got co-pilots and we got some cloud code but like how do I get to the next level and I think at the core of it it was figuring out how we distill I would say it was a disagreement but it's really how do you distill it down in a more simplified stepwise manner.

24:11 Yeah because in in you know we have we have great people from massive institutions come and contribute to the paper but really if yeah you gotta have a place to start and so I think that was the core of it. I think what like most controversial how to I don't think it was even controversial.

24:24 I think it was conversation. How do we make the paper I like the way I like the way you're arguing about what was the most we're both we're both Canadian for me. Yeah both Canadian right yeah yeah I think lending on terminology and lending on the on the levels to certain degree to make the pay because like yes there is lots of some people that have a lot of sophistications and they have financial reporting goals and they have you know auditability goals and that is great awesome there's a lot of that happening and they need very particular things that that cost lots of money and all of those things.

25:00 On the other hand you have a three-person startup that is starting to build things how do you make a paper that that contains principles that contains uh guidance that is applicable to both yeah and you can start at any level and you can make it useful and benefit from at least understanding to a large degree what are the things you should be concerned about what are the what are the types of things you should be looking for in order to try and get more uh safe and secure in what you're trying to do.

25:37 Cool so we're gonna wrap with that last question. So people are reading this they love it where should they start?

25:43 Let's start with you Marshall what where should they start on Monday morning what's the first thing they should do I think uh the way I put it is I mean going back to the kids I think that sometimes these are kids with chainsaws right like playing with chainsaws so I would basically consider like think about what is the chainsaw you're using.

26:00 Like is it for example do you already have like a container uh environment for like you can actually run your agents like if you don't have like cover the basics cover the basics of like you know not exposing secrets in plain tags not like letting your the agent actually like figure out things that are that are not really shouldn't be easy to figure out so I would say cover that get the egress going like find like control points where you actually can start to more safely and responsibly play with uh play with this technology and kind of keep evolving as maturity so that's that's how I would think about it in terms of like where are you I I'm quite far away from you here much but it sounded like

26:32 you just said be careful with the chainsaws your children are playing with because that was did I hear that correctly okay that's not an analogy I've heard before but I love it.

26:40 I love it.

26:40 Ian same question I mean I think the question is okay are you even at level one which is like do I have a common setup for people inside my organization to use agents whether in the cloud or locally that they're all sharing and that that's a that's the beginnings of a system right I think that's the first step is what we know about this entire space is it's it bet it doubles down a lot of the best practices of the last generation which is the more consolidation you have around the same workflows the more consolidation you have around the same context sharing around the same core systems the better these things are that's both from a security standpoint but it's also deeply from a usability standpoint

27:18 the amount of value you can extract. And so I would start there. And then I would say okay how do are we there? And if we're not how do we get there and then how do then then you get to the evolution of like okay let's go from level one to level two oh and eventually we want to have this you know reactive proactive factory that is you know driving the business I would I would just start with like it's almost like a self-reflection and understand what level of maturity you are where you are more mature where you're less I think I think for most people listening to our customers is they've gone out and they've bought seven different coding agents and they have cloud agents, they have local agents

27:49 there's this buffet in their organization. But the truth is in order to make this stuff really hum you kind of have to go through the exercise to start to consolidate.

27:56 Right right I think I might be a little biased in answering the question. First totally agreed but I think fundamentally you want to start looking at containment so how do you put agents in some uh boundary where you can control uh that you can start implementing other uh other uh other aspects of the paper on like networking ingress or other components that are important because it is really hard to do so if the agents are not contained.

28:29 Also so as that call to action that you know hopefully after you've heard a lot of this most important thing go to agentbaseline.org take a read of that you know that that uh reference architecture that v1 please take a read and give some feedback give us your you know give us what worked what didn't what was missing let us know because this is going to be an evolving document and it's what it's you know what we need to be able to make it useful uh for the masses um there's also uh an agent baseline slack I believe right so uh how how how should people go to join that agent baseline.org and following agent baseline.org join the slack drop some comments drop some GitHub issues love feedback and

29:05 think it's open for a reason it's community motion for a reason I don't think we're experts um but this is an opinionated take and we're we'd love to see the feedback. Awesome so do check out that uh that document give us that feedback and uh for now uh Ellie Ian and Marcello thank you so so much for all your wisdom all your insight and uh thanks for putting together that paper as well.

29:24 Thank you amazing. Hope you enjoyed that thanks for listening. Tune in to the next episode soon bye for now. The AI Native Dev is brought to you by TESL the package manager for skills and context. Your hosts are Guy Pajani and me Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast it's a community and we host monthly meetups at the TEL offices in central London.

29:46 Visit Tesle.io forward slash community to learn more and I hope to see you there.