Maggie Appleton: Agentic Coding Has Outgrown the Text Box

0:00 I just feel like this like very textual, simple interface isn't cutting it. And I want agents in there proactively doing stuff. I do think agents prompting us more would actually help relieve our cognitive load, which a lot of us I think are struggling with in the age of agents. And you can't solve it with a multiple choice text question.

0:17 You're asking me to pick between a bunch of MPM libraries. Give me like a comprehensive report of how these compare. Don't wait for me to prompt you.

0:25 The AI Native Dev is a podcast with developers and engineering leads at the cutting edge of AI and agentic coding. Join your host, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

0:48 Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Identific code is moving fast enough to give anyone with that. This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs hacked in to hear speakers from our topic, OpenAI, Netflix, and GitHub.

1:09 Cut through the hype and get to what actually works. This November, we're bringing it back to New York. Big up, folder, and full of people actually building the agency future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.

1:30 As a podcast listener, you can get 15% off your ticket using promo code pod 50. We have so much planned, and we can't wait to have you join us in New York. Hello and welcome to another episode of the AI Native Dev. Joining me today is Maggie Appleton, a staff research engineer at GitHub Next, and we're going to have a great conversation tackling such topics as how humans and agents interact and what are the handoff points between them when we think about planning, when we think about development, and so on.

2:06 So we've got a huge number of topics and depth that we're going to talk about. Maggie, welcome to the podcast. How are you doing?

2:12 Good, thank you. Thank you for having me.

2:13 Oh, absolute pleasure. Now, Maggie, tell us a little bit about uh your role and what you do at GitHub next.

2:18 Sure. So technically, staff research engineer. That is the title of almost everyone on the team. I'm actually a designer, I'm a front-end engineer as well, but I'm not the best. I'm much more at home making like uh exploratory interfaces, visual design. I came from a background that was like an illustrator art director, and then moved into more front-end stuff working on like dev education.

2:38 Um, and then I I just love a lot of things like uh theory of programming and like researching it, what's the future of software development, which is why GitHub Next is a very good place for me to just kind of do interesting experiments about what GitHub should work on next.

2:51 Yeah, no uh I was just about to say what is GitHub Next, but for people for people who haven't heard of it, tell us a little bit about the kind of it's like a lab research space, right? It's super interesting. Tell us a little bit about a few of the interesting projects that are being worked on and that have come out of GitHub Next.

3:06 Sure. So yeah, GitHub Next is is the lab's RD branch. So like pretty standard like innovation. Uh what the usual product engineering org wouldn't work on because it's too risky or too far out there. This lab works on like big swings, uh further in the future sort of stuff. Uh and this lab originally made the first GitHub co-pilots, like the first ever AI coding agent before anyone else had made one.

3:27 Um and of course now there are plenty, but this was like the OG. I wasn't on the team, so I'm not taking credit for that. But uh there's tons of always tons of interesting projects. I feel like we're in an exploratory phase right now. So everyone's prototyping like mad. Every morning I wake up to like five new prototypes to try to understand and figure out if they have legs, like a lot of stuff's early.

3:46 Um but some of the big projects out of the lab at the moment is like um agentic workflows you had Don on a little while ago to talk about, essentially software factories, or like he's trying to lobby for the phrase software farms, which I like more. Like you're harvesting the crops of agents all the time.

4:00 It's less mechanical, it's more like growing.

4:02 Yes.

4:03 Um, and then I'm working on a project called Chopin, which is all around how we plan with agents and collaboratively with other developers and other members of your team. So not just you and an agent alone, but you and your whole team planning more rigorously together and more visually, because I'm big on like diagrams, visual explanations, like tapping into how graphic design can help people understand complex concepts.

4:25 So for me, that's a big piece of it. But other people working on the project are more excited about you know the multiplayer piece and and tapping into like letting all your local agents work with it. So there's there's layers to the project.

4:36 I totally thought you were gonna say, like, Chopin, it was gonna be an orchestration piece or something like that. But I suppose with multiplayer it's a lot of people. There's some in there.

4:42 There's a lot of stuff in there. Um my uh Shistov is the engineer on it, and he's from Poland. So Chopin's a very famous Polish um composer, so he loved the name because it's all it is about orchestrating agents. There's multiple agents in there, you can kind of hand off to implementation and see how the work flows from plan through to finished product.

4:59 So there's a few things to it, but the piece I love is the like the collaborative planning, the visuals, the like rich, rich artifacts to help developers understand what they're about to ask agents to do, and then when the work's done, trying to comprehend what the agents did. The two big bottlenecks, I think, in in current work.

5:16 Nice. So you're in the GitHub.next space. Is it GitHub.next or GitHub next?

5:21 GitHub next. GitHub next. Like the next GitHub.

5:23 The next GitHub. I presume, I presume you refer to to current, like you know, the mainstream GitHub is either GitHub old or GitHub current. Yeah. And then I in my head there's a GitHub uh prev, which is like legacy, working with legacy clients or something. If you say, if you say no, I'm just gonna imagine that anyway.

5:40 So that's we kind of call it.com.

5:42 Oh, okay.

5:42 Because it's like we we don't tend to work on dot com because dot com is like a big, a big thing, right? With a lot of real clients needed to stay the same. Yeah. So we tend to work on things that are like more prototypes on the side or might end up rolling up into dot com, but not directly on the surface.

5:56 Yeah, nice, nice. So, Maggie, one of the things I love is going through past comments that people have made. And and yeah, dragging up the past and and and and showing the dirty washing on the on show here. So one of your lines was nine women make a baby in one month. Uh and that's a great I love that logic.

6:12 I love the line. Um so concretely, what does a what does a team look like that's you know shipping five features a day?

6:19 Yeah. Uh so this line came out of a talk I did uh that was about a collaborative uh developer uh platform that we had made called Ace or Prototype that Chapan is kind of related to. And it was essentially just like Slack, but with um micro VMs for code sandboxes in real time and agents in there as well.

6:37 So like all of you in one uh environment. And this is now like happening a lot more with their products now, like Buzz from Jack Dorsey, and like even Claude getting into Slack is like kind of going that direction. This was maybe like uh six months ago that we had a prototype of in this direction. So it's nice to see other people agree with our theory.

6:53 Um, and the the problem we were trying to solve was that uh everyone's going really, really fast with their agents locally in their terminals or their local desktop apps. So everyone on your team is accelerated, but like scaling up one person doesn't lead to like good collaborative products because everyone's off on their own branches doing their own thing.

7:11 It's really you get carry merge conflicts eventually, or you get features that like other people didn't agree to or didn't get to put input on. So one person could build something and they didn't really consult like the designers or the PMs or the back-end people, and then they have to like throw away the work after they've built it because it they weren't aligned up front.

7:28 So this metaphor of like you can't nine women can't make a baby in one month.

7:32 Like it's not possible.

7:34 No. Unfortunately, unfortunately, I know. I would put labor. Um, but you know, it in order to do big things, big ambitious things, you all need to be aligned together as one. You can't split up the work amongst all of you and think that it's going to lead to some cohesive product in the end.

7:51 Awesome. Uh one of the other things I think most people will probably agree that humans uh are similar to this as well. But you mentioned that agents are shit at CSS. Um and I and I would say that's because they've learned very, very well from us being really bad at CSS as well. Uh but honestly, what what would you say the current generation of models are typically very bad at that we're not acknowledging as a as an industry?

8:16 Yeah, I feel like I I do hear people complaining about agent design skills as I think the term they would roll it under. Um and when I hear people say they're bad at design, what I'm actually hearing is models are trained on language, right? And and of course on code, so they're trained on CSS. But the way that a human designer like looks at designing something, they are mostly working with visual input.

8:36 They're working with what we would think of as like classical graphic design. So you are trying to say, okay, what's important for someone to see on this page and what do they need to do? And then in order to achieve that, you use things like space, contrast, priority, color, grouping. Um, you like use the principles of how humans interpret objects and colours and values in space to like make the person see the right thing in the right order, click the right button, understand what's on the screen.

9:03 Um and agents haven't been trained on visual inputs. Like we can do, you know, image generation and pass images to them, but they're really not good at translating like visual reasoning into code in the way that a traditional human designer does, right? Um, so sure, it doesn't help that the CSS is all probably pretty pretty crap in the training data.

9:21 Um, but I think it's much more this missing piece of that they're not multimodal enough, they're too heavy on language, and so they're unable to like see what we can see. So I often, when working with models, trying to prompt them to do designs, they will just like make elements overlap because they can't see, even when they take screenshots, they don't really understand how much space needs to be between things.

9:41 Like, what does it mean for something to be visually aesthetic? They don't have an understanding of aesthetics, especially because it's um historical and cultural in the sense that like what we think looks good changes over time. So there was an aesthetic in like 2010 that if you had a website in that style now, we would think of as very old school, like, oh, that looks like you sliced that sidebar out of Photoshop and like stuck it in, you know, like we used to do.

10:02 Um and now the aesthetic is like linear verselle, clean, kind of modernist, but that will change over time, so we can't even like bake into their weights like what good aesthetics are because that evolves.

10:14 Yeah.

10:15 But we might be able to do it through, you know, skills or just like giving them more relevant data for models at a certain time period, but it's like a a more fundamental issue with the fact they're so language-based and they don't have this visual reasoning capacity.

10:27 Yeah. Do you think do you think the visual reasoning will need to uh uh account for or cater for agents as well these days? Because more and more agents tend to almost be going to the browser to do things at times where APIs and things like that don't exist. Um is that is that something we need to cater for?

10:44 And I guess a follow-up question is in terms of what you see as fixing this, yeah. Do you think general models can will learn to do this, or do we need like some kind of specific model to uh I think we'll probably get general models there.

10:58 I assume that the labs are doing more multimodal training. Like I assume, you know, I mean, given the speed of things, maybe two years. Yeah, yeah. If it's slow, maybe five. I definitely am not in the camp of like, oh, models will never overtake designer jobs. I I don't know that they'll take their jobs, but like in the same way that we've been able to automate away large chunks of the software engineering process.

11:18 Well, I should say like specific tasks within that uh job in the same way. I hope actually there's a future where I can describe what I'm trying to achieve to an agent and the interface it makes is at least usable, like baseline aesthetic, there's enough space between things, the text is readable, there's not a purple gradient on it, like there's like some baseline that I think we're really near and within a year or two we'll get to.

11:40 Um and then the question about like, do you do you think like agents will need to browse visually? I feel like we're mostly just gonna redesign the webinar websites, and I know there's other ways they could browse, but I think we'll just they'll have APIs and they'll just like use agent methods because text will always be faster.

11:57 Visuals are faster for humans because our you know brains are built differently to, you know, I say agent brains, not the same, but the way we process information is different to them, and we like visuals and space and gesture and movement in real time, and they're very like static language, you know?

12:11 Yeah, just give me one line with no spaces, no white space, yeah, yeah. Um okay, one other thing that you mentioned is boundary objects, and and and you borrow this from Susan Lee uh Starr's sociology. Um so most engineers have never heard that term.

12:25 Yeah, I'm sure they haven't.

12:26 Tell us a little bit about what it means and and why um I guess agent MD files fail that test.

12:31 Yeah. So yeah, so this is like a classic sociology paper. Like I studied anthropology. This is like one of the things you read in 101. Um, and it's from 1989, it's not that old really, but it was the study of how um Berkeley's Museum of Vertebrate Zoology first got started, which was like early 1900s.

12:47 And these sociologists studied how many disparate groups of people came together to found this museum. So you've got academics, you've got funders, you've got biologists, you've got like trappers and naturalists, and they all had like different levels of uh training and expertise. They had different priorities.

13:03 Some people care, of course, more about funding and and rules, and others are just concerned about like biology and capturing specimens. But they had effective boundary objects to collaborate, is the theory that they worked with here. It's like they had specimens and labels and field maps, and everyone read these objects differently.

13:18 They like worked differently for each group, but they enabled effective collaboration. And you could point tons of examples of this out in the world. You just think about any huge project that brings together many different kinds of cultures and communities of people, they need effective objects to collaborate around.

13:34 Um, but the objects can mean different things to different people and take different forms for each kind of group that it interacts with. And this applies to to humans and agents pretty well. If we treat agents as a kind of being with a kind of way of seeing the world, and it's a very different to the way humans see the world and think, and we don't need to perfectly understand each other's worlds because that's probably impossible.

13:54 Um I always kind of refer to the fact that you know um we are humans embodied and we like talk in real time in space, we're very social, agents have none of these qualities. You can give them cameras and mics, but they're still not going to understand what it is to be an embodied being in the way a human would.

14:09 Um but they don't have to. As long as we have effective what what we currently use is right, yeah, agent MD files, skill files, MCP servers, effective boundary objects to collaborate with them, then we can actually achieve things together without needing perfect shared understanding.

14:24 So is that context like an agent MD? Is that an effective boundary object?

14:28 Uh for the moment it's sort of okay, but I don't think I think um it serves agents well because it's it's a text file. Everything that we do is like very text heavy. We are reading these huge outputs in the CLI. Even the fact that current agent coding tools are so CLI focused, I think is like preferencing the agents' preferences and not the humans in terms of understanding and being able to communicate.

14:50 Um so we are having to read and write masses of text, which is not very effective for humans. We are much faster visual, spatial, social reasoners. It would be much more effective for you and I to like plan out some complex architecture problem with a whiteboard in real time, right? Drawing diagrams, talking, pointing to things, moving around.

15:08 It's much less effective for us to like type words into a document and that to be our way of expressing what we want and problem solving together. Um so the the big thing I'm trying to push for is like we like there are fine boundary objects, like these text files, but we could make them preference humans far more by making them social, real-time, interactive, visual.

15:29 Um, and I when I say interactive, I mean like it's not a static text file. Like we have beautiful like systems like the web and browsers where we could make everything interactive. You could be able to like po poke around an interactive diagram, make a a mock-up of your UI and compare it to what you might want and move sliders and move little things.

15:48 I like in design we make jigs a lot, which are like little little control panels just to solve one particular problem. These are the sorts of things when I say visual interactive that I mean is like make this stuff easier for humans to understand in a way that a wall of text is not. It's very difficult for us to comprehend complex systems in walls of text.

16:07 And this kind of like leads on to the final quote, which you'll be happy to hear, uh, which is um when you described humans and agents as looking at each looking at each other through a keyhole. Uh and this is really interesting. It's kind of like that, you know, that problem of uh of really not being able to kind of like see as much or understand as much of the the the entire entity as uh from both sides as as as you might want.

16:30 Who whose job is it to, I guess, widen that keyhole? Is it the the model builders? Is it the tool builders like yourselves? Um yeah, who who who or is it the humans, I guess, learning better how to prompt, how to provide that information and context to the agents?

16:44 I think it's it's mostly tool build builders, I think. I would maybe include model builders in this in the sense of like actually the moment the model builders and the tool builders have so much overlap, and you if you think about the fact that Claude and and um chat GPT really dominate in terms of like they kind of own the interface and the model.

17:00 Um not to say that there aren't other players in the space, like of course GitHub Copilot is another one that like gives you tools but doesn't own the models. Um at least in terms of widening the keyhole that letting letting the agent see more of our world, uh at the moment it is like we're just like funneling text through this small hole.

17:18 Um I mean I can kind of touch on like interpretability is like the study of how we understand how models think, and that's much more what the model companies do, right? Because from the outside we can't do that. That is like foundation labs are really the only ones in a position to do meaningful interpretability work.

17:32 And I think that stuff is important to s for us to see how agents reason. But to get back to the boundary objects thing, it's actually not as important for us to understand their worlds more comprehensively, or even them ours, as much as it's important for tool builders to make these kind of environments that humans can do good thinking in and then create artifacts that you can pass off to agents that are optimized for the agents as well, because they want to, you know, they can happily r deal with reams of text, just tons and tons of files, like nested skills that just contain all kinds of things.

18:02 But that is really difficult for humans to work with. I think it's the responsibility of the tool builders to make better interfaces and environments for humans to like figure out what does the system does, what do I want it to do, like do how do I design a reliable architecture that will like stand the test of time?

18:18 Those need different interfaces to just like text editors.

18:21 Yeah, it's down to the developer experience or the human experience now in terms of how we actually how we actually interact. Um let's talk about planning. Let's talk about planning and design, I guess. Um how does that change when agents actually, rather than um I guess being on the other side of the keyhole where we're just providing information to them, how does that change when they actually become collaborators?

18:42 So um roles, responsibilities, how they help and how they become part of the team. Is that something that we want, or is that something that you know we actually want it to be much more human-oriented and actually maybe we get a bit of input from agents? What's the best dynamic you think we should have?

18:58 I definitely think agents are active participants in planning is really good. Because it kind of gets into not like you know, I think we should have a thousand layers of agents and everything. There's like the agent over here who might implement something, but they can be far away from the fact that like when you are trying to plan something out, there's a whole stage where you need to like understand the current state of the code base um and then figure out what you want to change.

19:19 And there can be all kinds of helper agents in there, and we already do this, I think, in a in a way that is inefficient. Where like we're in our CLI, we're like, okay, I want to plan something, like go into plan mode or whatever. And the agent's like, okay, I'll spend more time like asking you questions.

19:33 Like, I'm sure if you've you've probably used like Matt Pecock's grill me skill, like very infamous at this point, where it asks you like 40 questions in a row, all multiple choice in this text editor. And for almost every question, I feel like each could be its own like research thread or like rabbit hole I need to go down to say, like, okay, you're asking me to pick between a bunch of NPM libraries.

19:51 I don't know, give me like a comprehensive report of how these compare and how they relate to the problem I'm trying to solve, or like you're asking me about visual design, show me visual design options, like show me my design system. Do we need to change the design system for this design to work? Like, every one of those is a complex problem to dig into, and you can't solve it with a multiple choice text question.

20:11 You need to make it its own rich artifact. But this is where agents can help. Like every one of those could have a bespoke interface designed for you with agents, which is also why I care about agents getting better at visual design. It's like I really do want generative UI to work out, not for the end product, but for the interim developer experience.

20:29 Um, so like I just feel like this like very textual, simple interface isn't cutting it. And I want agents in there proactively doing stuff too. Don't wait for me to prompt you, be like, oh, I see where you're going. You're trying to work on something with auth. I'm gonna go research a bunch of like cutting-edge practices or like what are the new libraries you could be using.

20:49 I'm gonna go do web search ahead of time so that by the time you've reached this topic, I know we're gonna get to in the plan. I already have reports ready for you with bespoke interfaces ready to go. So a lot of I'm exploring this with Chopin is like how do we get agents to look ahead, be proactive, do the research ahead of time, present you with decisions like as you move through without you having to always be the one to like prompt it.

21:12 But I think people have been not controversial about this, like you know, get the agents to prompt you because it does think like, well, the agent sometimes forgets to ask you things or like it can't it's not as good at critical thinking as we are at times unless it's prompted to be. Um so you don't want to like let the agent drive too much, I think is the worry.

21:30 And at the same time, I do think agents prompting us more would actually help relieve our cognitive load, which a lot of us I think are struggling with in the age of agents. You're just constantly juggling seven different sessions and trying to remember what you've been doing. I really want agents to be like, hey, every morning, here's a summary of what we've worked on.

21:47 Because of these, here I think are five good next steps you should consider. You know, it doesn't I don't have to run with them, but I want them to prompt me more. I want them to be doing more sense making across the work I'm doing over the course of a day or week or month. And saying, you're n you're forgetting this, you haven't looked into this.

22:03 Good to think I looked into it for you. Here's my results. Like, I'm looking for that.

22:07 And there's a couple of errors I want to dig a little bit deeper in. One, which we'll get to, is the single player versus multiplayer on that planning. Because I think the dynamic changes and how you actually interact with the agent changes. But I want to touch on junior developers. Because this is I uh what I what I think this can lead to is actually a really good path for junior developers to actually rather than you know the worry is oh a junior developer hasn't been through the trenches like like a senior developer has, they don't know the pitfalls, they don't know what to look out for or or how to design or plan well.

22:38 Now, if you have an agent here helping with that planning and understanding the pitfalls, or at least having the skills or knowledge to recognise what a senior developer would do or the pitfalls that a junior developer might walk into, how much how much will will that do you think um kind of bridge the gap between the senior developer, junior developer uh wisdom that that sometimes lacks?

23:02 Hopefully a bit. I mean, if we think about like pair programming must be totally dead by now. Which is how you used to learn. It's probably still alive, but it's a pairing with an agent versus thing of like everyone like vibe coding their own apps, right? They don't even know what questions to ask, they don't even know to ask about security, or they don't even know to ask about how deploying something works.

23:22 Yes, yes.

23:23 Um but but if you ask the agent the right questions, you can of course get good quality answers that like teach you all about those very important topics. But I would hope like in when we get agents to move into these more real-time collaborative spaces when we're doing our development work and out of these like local silos.

23:39 I the hope is that juniors and like everyone on the team just gets more visibility into everyone else's agent sessions. That's like one piece of it. Is the way we've designed most of these systems is open by default within a team. So like everyone can see everyone else's agent chats. This is like we got, of course, feedback.

23:55 This is a bit controversial that everyone's like, I don't want people to see what I'm asking the agents, but if you're on a team, you're building the same product, you really sh it should be open.

24:03 And it should be sharing as well, right? Because you you want that level of uh you know one person not being super effective and other people being you know ineffective because they they they they don't understand the prompting just uh as much and so forth.

24:15 Exactly, exactly. Seeing how other people prompt the agent, seeing what kind of questions they ask. But even if you were able to like like I'm I don't do backend or I I if I do backend, it's very bad. But I get to work with people who are like very senior backend people. If I could actually see them writing plans in real time and seeing how what they correct, what questions they ask the agent, where they ask the agent to go do research, just like I think that's the way that you would get junior developers to actually learn is to be in that sort of environment of like they see the quality of the level of plans that a senior engineer would hand off to an agent and they understand more about what's

24:49 gone into it. Versus like I don't even know what they're doing at the moment, other than sitting alone with their agents not asking the right questions. Yeah.

24:55 And and I wonder, I I question even if if the junior dev, let's say, even needs to know some of those things. Uh as in, needs to know to ask the agent some of those things, or whether it's actually kind of like built into the agent that other, you know, the more senior developers are asking those questions or are uh you know offering um opinions in these areas, I should make sure this is also considered here.

25:22 Yeah. And I should probably tell the junior developer, designer, planner that this is what I'm doing, and this is maybe I you know this is an important area or something like that.

25:31 Yeah, I'd be surprised. I'm sure people have skills for this already.

25:34 Right. It has to be it's gonna be more a guidance or skills thing.

25:37 Yeah, or maybe they'll bake it into the models where like models now, I think that we're finally getting a handle on memory where it really has an understanding of your profile and what you're good at and bad at. And I think if it could recognize someone was a junior, I would hope the model was to some degree trained to m ask more proactive questions or like understand.

25:53 But it conflicts with the with the rule, like you know, the human's always right.

25:57 Yeah.

25:57 So I don't know, do we like put junior developers in a position where they don't have as much authority over their agent? That seems troublesome.

26:04 So maybe it was instead of saying you're absolutely right, maybe it would just say, Ah, you're you're probably right. Yeah, yeah.

26:10 I mean, this is like uh because for a while I was a bit worried about the whole thing of of agents always validating our ideas, and like I always I wanted an agent that would like enable my own critical thinking and my own reflection. So at some point I put in my Claude system prompt, like, you should question me on everything.

26:25 Like, don't let me just assume I'm right. But um, it was so extremely annoying the way that it behaved afterwards was every single thing I said, even if I was like, I'm researching new kitchen scissors to buy, it would be like, Well, you probably aren't correct about your assumptions of what you need.

26:42 Just I this turn off, never mind. I regret this.

26:46 I shouldn't have asked. Um let's go into single player versus multiplayer here as well. Um so as an individual, if I was to think about um planning, yeah, I you know, just interacting directly with an agent makes a ton of sense that I can kind of like ask some questions, have a good back and forth. But of course, let's say you and I were to do some planning, exactly what you said, we'll be on a whiteboard and we'll be talking and drawing and and and visually planning.

27:12 How does an how does an agent best interact there? Is it is it something that they should validate in post? Is it something that they should almost like you know hear our conversation, understand our assertions and and why we're doing things, and then give commentary? What's the best place to use?

27:28 Uh okay, the thing I'm playing with at the moment is is a little bit of them like watching quietly the chat. So uh in the prototype we have in Chopin, we have like a chat and a and a document, and the document is more like the the artifact you're making, the plan you're making, and the chat is of course free form chat.

27:44 But when we're when I'm planning with my engineer, like we are also usually on a call talking. And at the moment we don't have it hooked up where the agent like reads our transcript, but I do think that would be a good direction to go in. But at the moment, what I have is like we're typing in the chat, and of course, like you know, Jev is like the the hottest thing ever.

28:00 Of course, the minute Jev came out, I was like, perfect, we've been doing this badly with models, let's go. Everything you type in the chat gets classified as like, is this a question, a proposal, an opinion? Uh is this a constraint on an existing decision? So the agent is constantly trying to understand what's happening in the chat, and it's putting things into the plan in structured formats.

28:19 It's like marking like this is a decision you need to make. Here are the options you've laid out for that decision. Uh, here's like the goal that you've stated in the chat. So you're trying to turn this like very continuous free-form human conversation into a more structured artifact that you will work towards handing off to an implementation agent.

28:36 Um, so this starts to head in the right direction for me. Like, um, we're trying to figure out how to put in like TL draw whiteboards to be able to draw us.

28:44 Yeah.

28:44 And I need to figure out a way to hook up something like a real-time cool transcript to it. But my ideal like planning environment is free-form chat with someone where we just like hash out the details, but the agents are these background agents classifying, and then when they spot something like we have an open question that could be a web search they need to go do, I want them to proactively write that research report, put it in line in the plan.

29:07 So that's the next piece we kind of want to work on.

29:09 Yeah, really interesting. And I think the handoffs that you mentioned there, or when the agent interacts, is super, super critical. Um, as a parallel, as an aside, I've been I've been using Claude, I must admit, on voice uh on voice chat while I've been cooking recently. And it's been, it's, it's, it's not great at timings, uh, but it's good with it's good with telling me, oh yeah, you should add this or that.

29:31 I kind of like show at the fridge and I'm like, oh yeah, you can add this and this and this and add that. And one of the it's great when it's just me, but then when my kids walk in or my wife walks in and just say something, all of a sudden Claude's like answering back in my ear, uh, and it's like, okay, that's not great.

29:46 I don't want you to interact at this point. And I guess the one of the one of the questions here would probably be like recognizing there is a there is a point in time when I want an answer from you or I want a contribution from you, and it's almost like keep like milestones where it's like, okay, we've we've got to this stage.

30:03 What have we forgotten? What do we need to add? What are we what are what are the constraints should we be aware of? Yeah, and then get it to get it to come in at that point.

30:10 Yeah, yeah, I know. I this is why I've struggled with that real-time chat interface. Like I've tried it with chat GPT, but it's the same thing. It's like someone I'll be like out in the park walking with my kid and then trying to talk to it, but then of course, you like look weird like you're chatting to this age.

30:24 Someone gets too close or they come talk to you, you're like awkwardly like hang up a call with an age. I don't know, the whole thing socially doesn't work for me. Um, and it's this really I love it.

30:33 I love that. I really love it socially, yeah.

30:35 Yeah, um, but it is like I think the way I've always tried to design these like these collaborative like agents in real-time systems is agents as in the background. I don't want them interrupting, yeah. Especially if I'm talking to a coworker, yeah. Like they do not have the right to like come in. Do you know, like in the beginning when we had all these um agent transcribers in like Zoom and stuff, and they would like look like a person in the screen.

30:57 And I was always like, they do not have the prominence to be a person in this call, they are like a background helper. Um, so I haven't really explored much where the agent interrupts in any sense. I definitely want it to be like it's doing continuous background work in a way that you can always uh discard it.

31:13 Because it might do some research that you're like actually not relevant. Like my opinion, I already have experience in this, I'm picking this framework, I don't want to hear about your research report. I think it's something like we have to assume all the agent work can be like just disposable. Like it'll become cheap enough and intelligence will be cheap enough that this is all just like fine for it to constantly churn and try to suggest things and then we can dismiss as needed.

31:36 Um but I don't I don't I haven't yet figured out how they could really be active participants in a way that doesn't feel invasive, annoying.

31:44 Yeah you almost need like that JEV decision that the agent thinks, should I interrupt it or should I not interrupt it? Am I can I can I contribute something that the that might that the users here would actually find valuable at this at this point? Yeah um let's talk a little bit about handoffs, huge like human to agent handoffs and agent to multiple human handoffs as well.

32:05 Um I guess where where do you find what do what do you find is the biggest problems with handoffs today, maybe around content speci uh context specifically.

32:14 Yeah. Um it's interesting actually, we're getting someone to come work on Chopin who's really into like formal verification. So so at the moment, right, we all like write this plan and we're like, you know, you might ask the agent, like, uh, have I thought through everything? And like it's usually just gonna be like sometimes actually it does have good answers.

32:30 Like you can always try and sense check yourself. But it's it's almost certain, right? You hand this plan off, and then the agent always finds things during implementation, right? I always tell it, like, okay, you can make decisions as you go because you're going to need to, um, if I'm doing it where I'm not around.

32:44 I just want you to log them. At the end, I want like a really nice visual report where you summarize everything you've done and why you did it, and a list of all the decisions you made along the way. Um, because we just it's so hard to comprehensively cover everything up front. It always finds bugs or edge cases, or like, oh, we didn't consider what happens in this in this state.

33:02 Um, formal verification systems are supposed to help with this, where like you might take a uh a plan that you've made with a coworker and it's to your the best of your ability comprehensive, but then try and turn it into a formal spec where it really tries to cover every single possible use case. And that also means that after the implementation, you can compare it to the formal spec and like validate whether it actually did the work it was supposed to in a way that's harder to do in kind of more free-form like natural language plans.

33:29 So that's one interesting direction is like could you just like computationally verify that the plan was completed to a certain degree? And the uh yeah, I do think the other one is just like their reporting on the other side um helps with you figuring out like okay, this is this is what went wrong along the way.

33:44 Yeah.

33:45 And and how much does this uh kind of lean into shared understanding before delegation? Because I guess I guess you know, is that shared understanding should that be between multiple agents, many humans? Where do you draw that line between shared understanding?

34:01 Yeah, um, I think we've explored this a little bit, but probably not enough in the sense that I know in some companies, like you really want everyone to not have like seen or checked the plan, but to have consensus that before you hand it off, especially um, I think if you're being ambitious about your agentic runs, they can be really expensive.

34:18 If you're really like, okay, we've planned this feature thoroughly, like we want to run, like I mean, I love orchestrator agents, I know everyone does, right? But I hand always hand mine to a powerful orchestrator and I say you use you know less powerful models as needed to do the sub agents to implement and test and review everything.

34:33 Um, so you have your orchestrator who you know is gonna run the show. Uh, you want to make sure that you haven't planned incorrectly. You want to make sure the plan isn't like um when the when the run's complete and you've already spent the tokens, you don't want the designer to come back and be like, well, that doesn't fit into our system.

34:50 Or you don't want your infra person to come in and just be like, we can't deploy that for some reason. Um, getting your whole team to sign off on a plan, I think, is more of a heavyweight thing, but I imagine you would do in bigger companies. If you're not a small startup and you're spending significant numbers of tokens on like really good quality implementation and checks and stuff, you want everyone to be like, we agree this is the right thing to build, it's the right time to build it, PMs agree like this isn't gonna flop in the market.

35:15 Like I I think we'll get see more of this like formal approval up front before we hand everything off to implementation agents.

35:24 Yeah, and and so that's very much human kind of like agreement. Yeah. Do you do you see that as almost like almost spec driven, or do you see this as something that's kind of like a combination of other other pieces?

35:35 Because I know you know Yeah, uh the team, the GitHub Next team has has done some previous research before I I joined on spec driven development, yeah. Where they were like the spec's the software, like how is it different? I think that can become more true with the formal verification methods. I think with natural language I'm a bit more skeptical because it's like what is code except for precisely stated, you know, like ha rules of how software works.

35:58 And the spec is like one level up from that, but people will argue like that's what code is. It's the code is the spec. Um, which I I kind of agree with. Um but yeah, I do I do feel like we're just inevitably moving towards towards spec driven development. Like, how much is anyone really gonna read the code?

36:12 Again, as long as like the tests verify that it does what it says it does. I yeah, I I feel like we just move into interacting with these higher-level artifacts. So these like plans and these like um after an agent completes a run, what we would currently do in like a PR description, but we obviously need new nouns and like new primitives there.

36:32 Um, but some sort of report of after your all your orchestrator agent and all your implementation agents have done some huge run. It's like you don't want a line-by-line diff. Yeah, you want a really high-level understanding of what happened, and and you can also have agents, you know, verify that final report actually reflects what the agents did.

36:50 You can have many layers of checker agents all through this. Um, but you'll just be reviewing these high-level inputs and high-level outputs, and you won't have to worry as much about all the noise down below because we'll have figured out ways to actually make this quite rigorous and testable.

37:04 Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

37:21 But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

37:39 Alright, back to the episode. Let's talk about implementation and how we actually now build with agents. Um I I said I wouldn't quote you again. Uh I lied. I I think this is one of your quotes. Um, in a world of fast, cheap software, quality becomes the new differentiator. Was that one of yours?

37:57 I feel like everyone said that at this point. I don't know if that was me, but I've definitely said it. Okay, that's cool. That's cool.

38:02 We'll say it's one of yours. Um so quality is notoriously hard to be able to measure, I guess, and validate. Um or even reward in companies as well. How how do you get a team or an exec to um to pay for, I guess, the craft that they don't see in a dashboard when speed and delivery is becoming, I guess, one of the one of the more uh sought-after pieces, efficiency in in in organizations?

38:31 I think it'll happen naturally because I think before you really had to fight for craft for the time to finally tune something. But this was in the time where software was was compared to what we have now, infinitely slow. Just you would spend weeks planning something and then like months implementing it, even like a dashboard or something.

38:50 Which now, okay, you can bang out a version in an afternoon that sure is not gonna be as like um verified that it's like good quality, and there'll probably be lots of bugs to solve. But we're in the age where you can assign a bunch of like autonomous agents to just like run a bug fix program, like go try and use the software on a loop, find every bug, file a bug, fix it, have another agent review it.

39:11 Like, this is like the software factory stuff is really solving a lot of this quality control to it.

39:15 And what does debt what is debt now? Yeah, it's just tokens, right?

39:20 It is what is tokens allow down. Um I feel like craft, yeah, craft becomes the differentiator, but I do actually think craft is more within our reach now. I certainly feel far more empowered as both a designer and developer to like today than even six months ago. Um, that I can do much more ambitious things and to a much higher quality level than previous me could have ever done.

39:41 It's just I just was not as capable. We would all just make kind of like okay software, frankly, in the past. I don't think quality is degrading from what I see around. You could you can look at like people who like vibe up slop, of course it's bad. But really, if you look at like the big companies, it's not getting worse the software.

40:01 It's actually getting bug bug reports are going down. It's like things are shipping faster. We're going through a weird transition period where where execs are demanding more speed because that you have agent tokens, you can just build stuff faster. But I think they're quickly gonna realize like deploying agent tokens effectively towards like reducing bugs and also like leveling up the quality of your interface and like being able to explore more options and like user test them with more people and then figure out what actually works.

40:29 It's so cheap to prototype these days. I feel like we get faster insights into stuff.

40:33 Yeah.

40:33 Um I mean, I don't know how you measure it, but I hope someone can come up with some quantitative measurements for like how the quality of software improves, but I have no doubt it's happening.

40:42 Yeah. So if we if we look at what's speeding up and what's slowing down and where the bottleneck is, then the uh how something is actually built becomes cheap and it's uh probably largely defined by models and agents and and and and loops. Um the the sh the the I guess the the how we build something in terms of the more of the planning and the design is is likely more human-led or or certainly human signed off and uh probably a genetic assistance needed, required, used.

41:14 Uh but one level above that, the should we build it, I guess.

41:19 Yeah, yeah.

41:20 That that becomes the the harder thing because in a world where in a world where building becomes cheap and we can um and everything's possible, what do we do? Yeah. Who who's equipped on a team to really answer that? Is that the engineer, the architect, or is it is it more leaning into the PM?

41:36 Yeah, I think it's the PM. I don't know. It's like I think like PMs get a bad rap, but like I worked with some really great PMs where like they make far better decisions than any of the engineers about what we should do. Uh and it really is when I say PM, I just mean someone who actually is like doing thorough research on the state of the market, on like what other products are available, doing comparisons.

41:56 I like when it's someone's full-time job just to care about like are we building the right thing in the right way? They kind of do become the arbitrators of like what we should build. I think before you would have had designers and engineers more helping make those decisions because they could speak to feasibility more.

42:13 But now that you can be more ambitious and build more stuff, I feel like there's less chance for them to say no. In the past, it would be like, you know, wouldn't it be nice if we could build this? Was like most product conversations. And now it is possible for a small team to build m much more ambitious stuff that I feel like you really can go after the ideal version of things and not always we had to make the compromise and build the crappy skateboard and not the car kind of thing.

42:37 Now I feel like maybe you can't, you know, always build the car, but you can build something like the bicycle. I don't know what the metaphor is here, but you can build more ambitious things and should let PMs kind of like guide guide the decision making of what to build.

42:51 And do you see how do you how do you see those two roles like scope creeping? I guess let's call it scope creep, given that we're talking about PMs. But um how how do you see the the the the PM role given you know the the development role in the engineering world is becoming more accessible?

43:08 Yeah, yeah, yeah. Um I know there's this thing of like we won't need developers, the PMs will just like prompt it.

43:14 Or is it or is it that the engineers actually become more of the PMs as well? Is there a merge happening there?

43:20 Maybe. I still feel like you need distinct roles. Like, unless you're gonna say the engineer's gonna become really good at user interviews and market research and product comparison and strategy, I feel like that's too much to burden one person with. We always want to try and collapse roles in this industry, but the engineer still has to be in charge of like, is this architecture scalable?

43:37 Are we gonna like fall over if a hundred thousand people like join at once? Like, I think you still need someone who's in charge of owning the engineering and like keeping really high quality product. And then that is a totally different job to use a research, market research, product strategy. And I think it it's like would be b a bad idea to collapse them because it's too much for one person to manage.

44:00 And they have, I suppose, different outcomes in terms of an engineer would be more focused on the quality and the and the and the architecture and the design. Whereas the PM would actually care much, much more about the why should we build this and who's this for and focusing more on the user and the outcome of the product.

44:14 Yeah, and it's the same with designers. Like designers aren't dead despite everyone going on about it. Like the designer still advocates for like what's the best interface for the user? What do they need to do? Like, how do we make this look and feel the right way? And that is again, I feel like a completely different skill set to what the PM does and the engineer does.

44:27 Yeah. Yeah.

44:28 Okay, so so let's kind of combine a couple of these and ask a ask a little bit of a teasing question here, which is if we could start our careers again, but in today's world, um, you know, we're we're maybe just finishing university, coming into uh coming into the world of industry. What kind of role do you feel given where AI is today?

44:52 You would pique your interest in terms of in terms of you know you being you wanting to be a builder, what would your role be?

44:59 Oh, I mean I've always been a prototyper really. Like from from early on, um trying to like build websites like as a as a teenager. I was always just trying to do like weird scrappy ideas. And I think I would have I I would love to be like 21 right now. Um because because I would always come up with these ideas and sketch them out or like design them in Photoshop, but like I could never take them much further.

45:24 Yeah.

45:24 Now I would just be flying. I would frankly, I would get myself into whatever a junior role or whatever company would give me tokens. Tokens are like the currency. I would get paid nothing as long as you give me a pay tokens.

45:36 Tokens, yeah.

45:36 Pay me in tokens.

45:38 Honestly.

45:39 And then just like build, and I'm I mean, I am doing this now in the time that I do have. Just build as much as possible. I think the lesson I keep learning over and over is like whatever idea I come up with, make it more ambitious. Like, I've never in my life done things like build like an eval harness, and like I'm now just being like, right, let's just try.

45:56 I'm just gonna plan it out with the agent, I'm just gonna attempt. And in attempting, I learn so much, right? And in failing and doing it for the fourth time over, I'm like learning so much each time. I would just get somewhere where it's like you have generous token budgets and you're just willing to put out as many prototypes as possible.

46:13 Not really trying to make them that robust or not trying to turn them into products. I don't know. I think there's people who try to like take everything and be like, how do I make this into a business? Maybe that's just not the kind of itch I like to scratch. I would just be trying to make as many small usable things as possible in order to learn.

46:30 It just introduces you to the full spectrum of engineering. Like I've I always had to stay very narrow on front-end and design because there's so much to learn outside of that. But now I can just be like, well, let's learn about what database I'm gonna stand up. Like, let's learn about how I'm gonna deploy this.

46:43 Like, I'm the I'm coming at it from the complete opposite angle, actually. Because I uh yeah, my background is much, much more on the back-end engineering. So and I was always terrible at design and front end.

46:52 Now's the chance.

46:53 All of a sudden, I I'm I'm like, oh my gosh, so I'm I'm I just realised I'm a front-end designer and I'm a I'm a front-end engineer now. Um and it just unlocks, right? Now to ask the question which I hate to get asked, but I love to ask other people, uh, which is the the getting the crystal ball out and trying to foretell how how the future will go.

47:12 In a few years, let's say, because I hate you know, years ago I would have said 10 years, but uh in a few years, let's say let's say let's say six or seven years.

47:20 Okay, okay, hard. Okay.

47:22 How do you see that role that you just mentioned?

47:25 Yeah.

47:26 Um, I guess, not the on entrepreneurial, but the the the creative, the the rapid application developer, the trying to trying to build those types of applications. How do you see that standing up to the test of time of just five, six years?

47:41 I would hope I would hope in the process of building lots and lots of stuff and being ambitious because agents allow you to, you would learn enough that you should be becoming very powerful with agents in a way that I think those of us inside the AI bubble forget how slow technical adoption is outside of it.

47:57 Um like because we're going at warp speed. But if you go talk to most people around, they're like, basically no chat GPT, like now they're hearing a lot about the AI safety concerns, like like the bubble, like things are filtering out of the bubble. Um, but still I feel like if you got started early and were very ambitious and really understood the capabilities of what models can teach you and allow you to do, you're still way ahead of everyone else.

48:18 I think it's not like those people would become that their skills would become like useless because the models advance so much. I don't see it going that direction. I don't think at any point models will be able to like read human society and culture and like make decisions about what the right thing to build is.

48:36 I don't know if it would ever get there. Not in the way a human really could. So it's more like a human who can like look around them and say, What problems can I solve in the world? is always really gonna have something valuable that the agents can never have.

48:48 Why do why don't you think an agent will be able to make a good decision on that?

48:51 I don't think they have access to the right inputs.

48:53unknown Okay.

48:54 I feel like they are not part of a culture and a society in a way where they can see what the problems are and also the fact they don't have physical embodiment. I know that's coming people say it's coming soon with robots and cameras and stuff. But even so, having a robot present in a social situation changes it in a way that like they I don't know, they just like can't go out in the world and be part of companies or or events or social groups in a way where they would see the same social problems we would.

49:20 So do you think it's not a it's not so much a model capability, it's a it's a it's a data problem.

49:25 Yeah, yeah, they don't have the right data. And I don't think you could ever scrape it. How how would you scrape and you just think of like what are really successful businesses? It's like they understood there was an unsolved problem for a group of people in a certain context. I don't know how you get the data of like being in a certain place and realizing like, oh, these people really need like for a simple version, like there's no bakery on this street where lots of people would maybe buy from it, or like I see there's some problem where like uh people don't have access to to like water in this town, like that even that level.

49:57 Um you need a human on the ground with social connections. If you think of like how sales works, it's all like social relationships, it's all like who knows who. It's like who who convinces who to buy something. Models I even though they might get at social engineering, they're never gonna have the social relationships you need to actually build a real business, like solve an actual problem for other humans.

50:18 Um, that's the piece that models can't replace. But a human with that context, with like I've spotted a problem, I understand how I might solve it, I now have can leverage agents to actually solve that problem. That's that's actually gonna be someone who has a lot of power in the world. And power in a good way, not in a like get all the money way, but in a like solve problems for people sort of power.

50:38 Yeah.

50:39 Maggie, we've we're at time. Um I wonder if we can get nine technical people to to record uh a podcast episode in five minutes. Just to just to wrap up a nice, nice little ending thought. Um Maggie, this has been really good fun. I really, really appreciate your time um and and insights as well and wisdom.

50:57 Where can our listeners go to learn more about Chopin as well as uh GitHub Next?

51:02 Sure. So Github Next.com, we have a little write-up on Chopin, just like a very early preview of what we're working on, and we'll put more updates on there. Uh and then I have a website, Maggieapleton.com, where I put up writing and research and everything I'm working on goes on either the GitHub Next site or my personal site.

51:16 Amazing. And where's where's where should people listen on socials for for for you? Or are you are you not a I'm on X. You're on X.

51:23 I'm I live on X. Let's call it Twitter. It is Twitter. It was it was Twitter from the beginning. Yeah. I live on Twitter. I love Twitter. I had like a dip, but it's still the best place to l like AI news, AI people. It's great.

51:35 Amazing. Maggie, thank you very much for for joining us today, and it's been a pleasure to chat with you.

51:40 Yeah, thanks so much.

51:41 And I hope you enjoyed that uh listening as well. And thank you very much for tuning in. See you next time. Bye for now. The AI Native Dev is brought to you by Tesla, the package manager for skills and context. Your hosts are Guy Pajani, and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community.

52:00 And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io, forward slash community, turn forward at the