Maria Gorinova: Feedback Loops That Fix Your Codebase

0:00 So we noticed that agents tend to become crazy with comments. The more you use the factory, the more time it passes, the longer the comments become. You need to shift your mindset to thinking of a software factory as something that your code, context, and data is part of. It is not a separate thing, it's not a separate entity.

0:23 It's not a some sort of like magic black box that does stuff for you that you can get angry at when it doesn't work. No, your code, your context, your data is also part of that factory.

0:37 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI energetic coding. Join your hosts, Glyfer Germany, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

1:00 Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netflix, and GitHub cut through the hype and get to what actually works.

1:24 This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. As a podcast listener, you can get 15% off your ticket using promo code POD50.

1:48 We have so much planned, and we can't wait to have you join us in New York. Hello and welcome to another episode of the AI Native Dev. And today we're going to be talking about all things agents, harnesses, and specifically loops, and in particular, feedback loops. We're going to talk about what they are, how you can create them, and how you can actually benefit from feedback loops in your organizations.

2:15 Joining me today is Maria Goranova, who is actually a repeat guest. So welcome back, Maria. How are you? Thank you, Sam, and good to be back. And last time we spoke, you talked to us uh about abstraction adherence, which is how agents uh use the right libraries, which was super interesting. But things have moved on, oh my gosh, so much more.

2:35 So much. And it was in this very office that I heard you speak about all things uh loops, feedback loops, and software factories. And in fact, one of the opening questions that you that you asked the entire group was what is a software factory? And I think it means different things to different people.

2:52 Um I'd love to I'd love to hear from you. What does it mean to you? And maybe even, you know, how does it how did it differ from person to person?

3:02 Uh the takes were very different from each other. Yeah, the responses for sure, everybody had a different concept, everybody had a different angle uh at which they were looking at it, some were thinking only about automation, other were thinking about particular designs, about particular architecture.

3:19 Um to me, I think uh the best way to describe what a software factory is kind of like go through history really of how agents have changed uh in the past two or so years, um or less than that.

3:36 Um yeah is a long time.

3:38 No, it's not a well, it's two years, is it? Uh I don't I I don't know. It it feels like it's been decades, but actually it's been such a short time. Um and I think you know, software factory is something we have been building towards in those many decades. Yeah. Um so um to me, a software factory is something where we have repeatable workflows, we have shared execution environment, and we have feedback loops.

4:10 Uh and feedback loops are ways for the factory to maintain itself, to improve itself, to make sure it's operational and it doesn't degrade into quality. Because once we go into this world of everything is done by agent, we are like we are supervising the tasks that that go into the agent sometimes, but we don't really supervise the code.

4:35 We don't really go into the details of writing code or even reviewing code at times, things deteriorate very quickly.

4:43 It's interesting you didn't mention ticket to PR. So someone raises an issue, a PR is created. But that's kind of like a use case of the factory rather than what the factory is, I suppose.

4:53 Yeah, I think that is one single workflow, one single automated workflow in a factory, which yeah, happens to be yeah, I want to write a linear uh issue, or I want to write a Jira ticket or something of the sort, and I want this to be then implemented and the PR to be opened and even merged automatically.

5:12 Um But there is so much more to this, uh especially again when it comes to maintenance, when it comes to um improvements that are not necessarily you don't need necessarily a human for them. Instead, you just churn through a lot of data. Well, you the agent, the agent churns through a lot of data, and then it can suggest improvements and implement improvements that make the factory better and keep it operational.

5:38 Yeah. As part of your session, you went through a brief history of of agents and and you know the various levels that they went through over the last what 30-40 years maybe? So you start you started with V1, just an agent, no context, um, or sorry, rather, no control uh over the the their performance.

5:57 Um and unless of course you own the agent's code itself. Uh what were the big bottlenecks before context with with just an agent?

6:06 I think I think you just said it in the question basically. So um if we only have an agent that we don't own, right? So maybe we are using cloud code that is closed source or some other closed source agent. We don't really we can't really do anything to improve the behavior of that agent if we don't have context around it, right?

6:30 If we if we just if we're just prompting it and that is all that happens, we there is no much control we can um exert over that agent.

6:41 So so an agent in a f agent without context in a factory, for example, uh from an autonomy point of view, from a trust point of view, it's very unlikely to do what we want it to do, so very unlikely to actually be of any use.

6:53 I think so, yes, because everybody has a different code base, everybody has different requirements, everybody has different taste. So you can't really have one agent that simply based on a prompt without anything else around it, um, can behave in a way that it's compliant, um high in quality, uh, you know, things things like that.

7:17 And and and then you moved on to V2, which is agent plus context. Um what did context really unlock for us then? And I guess what what's still missing from that?

7:28 So context by context here, I mean things like at the beginning it was you know, the Claude MD.

7:34 Yeah.

7:34 Claude and D, then it was agents and D, then we uh we had skills, uh also hooks are also a type of context. Um, all of these sort of documents that lie around in your repo, especially the ones that are specifically for use by agents. Yeah. Um, and uh I think what that unlocked is that now we have control over our experience inside of a particular repo.

8:01 You can share it with your team, right? Like you if you have Cloud MD, you can have your personal one, or you can have one that is in the repo that is shared with the team, and it kind of like defines how this team wants an agent to operate in that code base. Um, and of course, you know, with skills that becomes more sophisticated because now we you have different workflows that can be for uh called and so on.

8:25 Um I think what is missing is that here still we it's very it's all very manual, um, and there is no good way of evaluating this, right? Like, how do I answer the question of is my CloreMD good? Other than vibes? There is there is no really established mechanism for doing this. And also, if I wanted to improve the CloreMD, I I would just go and manually kind of like vibe something in it, or at best ask my agent to vibe something in it based on what more vibes.

9:00 It's just there there is no really principled.

9:03 So I guess we have evals, perhaps, which is maybe more of an offline test, more of a static test, perhaps. Um, but then perhaps this is where harnesses kick in and loops. So you you mentioned a harness as the agent, the context, and a feedback loop. Tell us what that actually now unlocks.

9:23 So a feedback loop, the way I think about it is really it's an agentic workflow, like everything else in this concept. Um but it has three distinct stages. Stage number one is about analyzing some data stream. So you provided it with some data, let's say I don't know, the comments that people have given uh to PR to PRs.

9:52 So I I take all of this data and I give it to the agent and I ask the agent to analyze it. So the second step here is um finding, based on this analysis, finding problems, potential problems or potential um inefficiencies, uh just things that can be better. Um in this particular example, we are if we're looking at PR reviews, uh maybe we see that the same PR comment is given over and over again to um to someone's PRs or the or a factory P uh factory.

10:28 Like a code review or something like that? Or yeah, like a code review. It could be anything, I suppose.

10:32 Yeah, it could be that there is always, for example, uh there is a particular way uh developers in this repo want uh a class to be written or documented, and every time the agent doesn't do that, and every time the developers need to comment on it, right? Something like this, just a made-up example. Um so identifying this problem in the data, and then the third stage will be suggesting a fix.

10:59 And this fix could be a code fix, it could be a refactor, it could be a bug fix, or it could be context improvements. So in this case, uh that I mentioned maybe uh the repo can benefit from a doc a doc that actually describes how we want a class to be defined in that repo.

11:19 And and I guess from the from the data source point of view, let's start with the data source. Um, what makes a good data source? Does it matter if it's a human data source or uh or uh you know a data source which is actually coming from a set of logs which are generated from uh an another agent or or another app or something like that?

11:38 It can be anything. It doesn't have to be human generated, it could be um the agent session logs where we can see um agents. So one thing that we have seen with in uh our internal dark factory, for example, is that agents are able to analyze their own session locks and see places of friction where they lose many turns to figure out how to use a tool, for example.

12:05 So that would be a potential place for improvement where the agent can now have some context that helps it not have to lose those turns. Um But I don't know if I can answer what makes a good data source.

12:20 It'll have to be a bit uh It sounds like it's anything that anything that displays, concisely displays friction or an or really has a kind of like a root cause of something that can be improved. Is that is that fair?

12:34 I I don't know if you can give it a definition like that, because uh yeah, like it's not obvious. Sometimes it's not obvious how a data source will with the agent session logs, I didn't know that will happen. That was kind of it was cool that it happened, and in high sight we can explain it. But um it's not that uh I thought, oh, let's introduce agent session logs because it's going to identify exactly these kind of problems.

13:03 Yeah, yeah. So the data sources we should just try multiple data sources.

13:07 Yes, I think so. I think it's a it's a matter of experimenting and seeing what comes out of it and and um yeah.

13:16 And what judges the analysis? So you talk about the analysis, it's it's essentially trying to identify issues that repeatedly occur. Um what does that? Is it is there a is there a specific agent that we use for that, or is I know Jeff has just interestingly come out. That's possibly an option, I suppose.

13:37 So it it depends. Um in this particular case, I was talking about feedback loops that kind of receive data over many, many changes, like many PRs, for example, many agent sessions, and so on. Um and for that you you just it's the feedback loop is an agentic prompt, and the agent you know works through that by looking at the data and doing certain steps and figuring out things.

14:05 But we also have other kind of feedback loops that are more tightly scoped to an individual change, uh, which uh we call the inner feedback loops. Um those ones, for example, we can have something speaking of Jeff, something like verifiers, so verifiers allow you to um define a bunch of criteria that you want a change or your code to meet, and then use an LLM as a judge approach to verify if that criteria is met.

14:42 And you can use Jeff for the for that, for example, or you can use another model.

14:47 And why would I use that over what a code review already provides?

14:55 I think it's um I wouldn't say you should use it over what code review already provides. Or as well as, rather. I would say as well as, yes. Like it it is um it is an additional dimension to it because code review is a little bit more open-ended.

15:09 Okay.

15:10 Um, a little bit heavier, I guess. Um even if you do it with an agent. Uh if you require human time, then it's obviously much more expensive than running an LLM as a judge. Um but even when you run an agent, like it takes a little bit of time, you know, it's open-ended. Uh verifiers are very tightly scoped.

15:29 You've defined specific rules that you want your code to follow that are not something that is easy to define with tests or with linting. Um it is a it is a form of linting.

15:41 I would I would actually classify it as a deterministic in terms of the triggering of the verifier?

15:48 Uh yes, we do trigger the verifier on every change. So when when a PR is up during um CI, we would we would trigger the verifiers. But the output is I wouldn't call it deterministic because it's still done by LLMs. Yeah.

16:04 Yeah. Okay, got it. Um so you talk about the inner loop. Uh there's also kind of, I think there's middle loops and outer loops as well, right? Talk us through uh what the other loops are as well.

16:15 Yes. So uh just to uh to clarify, so the inner loop we define loosely as something that operates on a single change to the code base. The middle loop is more like the examples I was giving before, where we have many, many changes to the code base and we get some data from them. We get the agent locks, uh the PR comments, but like from all of these changes, so that we so that the agent can, in a sense, generalize by looking at many, many examples across many sessions.

16:49 Um so that that is how we again loosely defined Biddle Loop. Outer loop is more about production signals.

16:58 Okay.

16:59 So if you want, for example, self-heal. Uh if the service that is running the dark factory fails at any point, we want it to be able to detect that it failed, see the error, and then self-heal um by fixing that. So that would be like an outer loop. Another one would be if if we have um a certain type of customer um signals, uh, for example, I don't know, um daily active users, yeah.

17:31 Right? If we see that going down, we want something to investigate what's going on and find the uh find the issue. Um so that would be another outer loop.

17:41 And what are the triggers for the middle and outer loops? Uh the the inner loops is obviously the trigger is the thing that happens, maybe it's a pull request that gets raised, so the loop runs on that. Um is it just scheduled for the middle and uh outer loops?

17:55 It depends what exactly you need and what your data is and and so on. Uh for most middle loops that we have, uh the trigger is a schedule, yes. So we run them every day or every week, um, sometimes uh every month for the very heavy ones. Um and they just they will just continuously kind of monitor um the code base and make sure that everything is fine.

18:22 Yeah. Um the outer loops more they're more likely to be triggered on an event, as I said, if the service is failing, or if some metric is going down that we want to go up, something like that.

18:35 Cool. Um I'd love to also, I mean, the the the picture looks wonderful in terms of these loops, it's self-fixing, um, but I presume it doesn't always work perfectly. What where are the kind of areas that if someone were to create a loop for the first time, they might expect it to go wrong? Perhaps it's the the identify the identification of maybe it's the data source actually, then maybe it's the analysis or the judging on the on the data to be able to perform that that change, maybe it's the change itself.

19:04 What what are the kind of areas that you would say are the hottest, the hot spots for for problems?

19:10 So if you wanted to create your first feedback loop, I would say the most important thing is to be very clear with your intent. You need to be very clear what the data source is, what you expect um or what you want the agent to look for in that data source, and what do you want it to improve based on that data source uh or based on the findings from the data source?

19:40 Um if you're vague, I think that is that is probably the the place that will be a hotbed for disaster because if you're vague, if you just say, Oh yeah, look at my past PRs and improve my repo. This is not this is not clear intent.

20:00 It's trying to fix everything and it actually needs to fix a very specific thing.

20:03 Um but also it doesn't the fixes are not going to be deep. There's not there there'll be a lack of depth and it will just make up things that maybe are true but not important, that that sort of thing. You want to create depth by really clearly explaining where uh to look for problems. For example, simple uh middle feedback loop we have is um one that uh we call comment diet, because we notice that agents uh tend to become crazy with comments.

20:38 They would just write a lot, like lots and lots of comments, and it the comments become longer and longer because the next agent sees long comments and then it thinks it needs to write. It is honestly it's mad. We've had some crazy files uh lying around. So the comment diet feedback loop would go and just the data source is very simple, it's just the code base, there is no additional data source.

21:02 So just look at the repo and run some scripts to kind of um count how many lines comments versus code there is, identify hotbed files that have too long of comments, that the comments are too long, uh, and then fix that. Just reduce the reduce the lines of comments in a way that doesn't degrade performance, right?

21:26 So this is more concrete. Now I'm like I've I've reduced it to something very, very specific, and it's not going to touch your code, it's not going to touch your architecture, and you know, big changes like this. You know that it does just that that one thing and it's focusing all its attention to do that one thing.

21:43 And then you can define a separate feedback loop to, I don't know, look at the architecture or the design or something else that you want.

21:50 Yeah. So in terms of feedback loops there, would it would it be is it fair to say there's almost like a couple of feedback loops there? One one which is maybe an inner loop. That when changes are made, comments aren't too verbose. And one or rather, you know, that that that's a that's a maybe a skill um point on the inner loop.

22:12 And on the on the middle loop, maybe that's more of a scheduled thing, doing an audit essentially of your code base to check that that's that that's okay and that's happening. Is that is that the right way of looking at it?

22:23 Um I think for some things you can divide it this way and have both inner and middle. Um I don't know if outer will make sense here, probably not, but um yes, you can have inner and middle. It depends on whether you expect this to be a drift or not, this this kind of behavior. Because so in this particular case, we notice that the comment bloat is a drift.

22:48 Yeah, the more you use the factory, the more time it passes, the longer the comments become. And yes, we can have some gates on the inner loop level um that check if the comments are too verbose and stuff like that. But I my suspicion is that even if we have those gates, it will still happen over time.

23:07 And and just to find what drift is, is that is that uh is that is that a difference between what we expect to happen based in the context and the reality in the in the code base good question.

23:20 Um I would define drift as some drift from some ideal state that that we hope for. And uh drift happens in human-maintained codebases too, like even before agents, you know, even before all of this. Um when many many developers are contribu contributing to uh a code base, over time, small errors compound, and then at one point you have to do a refactor or you or you discover a big bug or something like that.

23:54 Um there is uh a very good example uh for this is actually the rule of three. You know the rule of three?

24:00 Tell me.

24:01 Well, if you uh if you have to write code once, obviously you write it once. If you have to write it twice, you write it twice twice. But if you have to write it a third time, now is the time to introduce an abstraction. Yeah, right. So it's kind of like this like uh humans do it, agents will do the same.

24:21 Like you're going to write the code once, you're gonna write it twice, and then on the third time you need to uh introduce an abstraction, but that's not the job of the agent writing the third time code, that's a job of something else, something beyond that that is maintained. That even recognizes it's been done three times, whereas the agent won't necessarily exactly so it is kind of like it's something beyond, so that's why the middle loop looks at many, many sessions, because it it needs the bigger picture, it needs to generalize and correct those drifts, correct any spaghetti codes, yeah, as I like saying, um and um kind of you know fold back the repo in a nice state, in a better state that

25:04 is healthy, um, that subsequent agents will have a good time working in.

25:09 Yeah. Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

25:27 But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

25:45 Alright, back to the episode. Let's get practical now and and and in for for people who want to get started. Um, one of the things that you kind of like mentioned is is getting the correct mindset. What's the correct mindset for creating loops and thinking about loops?

26:03 You need to shift your mindset to thinking of a software factory as something that your code, context, and data is part of. It is not a separate thing, it's not a separate entity, it's not uh some sort of like magic black box that does stuff for you that you can get angry at when it doesn't work. No, your code, your context, your data is also part uh of that factory.

26:32 Um just to give an example, imagine you have uh project A and project B that semantically uh do exactly the same thing, right? When you run things through them, it's exactly the same behavior. But project A is written well, like the code is well structured, has good abstractions, uh, there is lots of context available, there is you know agents and the uh skills in it, um, there is uh I don't know, hooks that configure the agent.

27:03 There is like all sorts of mechanisms that um control this agency uh behavior. Um and project B has none of that. The code is looks like spaghetti. It's the same code semantically, but it looks like spaghetti. It is obvious that an agent will perform better in uh project A, right? Very much like a human.

27:25 A human will also be more confused by the spaghetti code than by the well-structured code. So this is what I mean by this. Uh you have control over your code, context, data. Uh, you can use all of this as part of the factory to make the factory better, because the factory is also your context code and data.

27:45 Okay, cool. And I guess I guess another thing is there are a couple of different types of loops that we talk about internally here at Tesla. We have a regular loop, and then we have a feedback loop. What's the difference between the two?

27:57 Well, I guess for feedback loops, we already talked about them. It's that idea of a loop that analyzes some data source, finds some problems and fixes them, or rather um requests fix fixes for them. Um what we call a general loop in Tesla is more of a any any sort of workflow, any any agentic workflow um can be a loop.

28:26 So um we have for example loops that um define the issue to PR, what um what that looks like. So how an agent should look at an issue, read the issue, uh plan the change, implement the change, and open a PR. Uh we have loops that look at PR review and can respond when there is a comment uh on on a PR, can then trigger um um and you know, either implement the comment or maybe push back if they don't agree with the comment, that sort of thing.

29:03 Um and feedback loops are type of loops.

29:06 It it almost feels like a the definition of a general loop isn't actually a loop, but I think you mentioned it was a workflow. It doesn't feel like it loops back on itself, whereas a feedback loop truly does feedback, it loop it loops, you know, takes information, passes it back, and and actually does a loop.

29:24 Yeah, it is a little meta, I think, because when we say yeah, it I think naming is hard. I think when we when we say just loop, even in those workflows that are kind of linear in a way, um but I like I think we are referring to the loop of the agent inside it, right? Because this agent is in a loop. This agent obviously, you know, uh loops through um many steps and tools and decisions and so on.

29:52 Um so that's what we mean by loop. And the feedback loop is like, I don't know, a loopy loop, I guess. The loopiest of all the loops, yeah. Like it. Oh no, there is more loopy loops. Yeah, there is there is there is the meta feedback loops that now would uh would analyze data uh that kind of tells you if a feedback loop is doing well, and then make changes to that feedback loop in order to improve that feedback loop.

30:21 So now we're managing the loops, essentially, right?

30:24 Yes, yes. Now we are looping instead of a loop instead of a loop.

30:27 Yeah, and I suppose that could go up any number of depths, I suppose.

30:31 Yeah, yeah, it's cool, no? It's it's a it's recursive.

30:34 That's really interesting actually, because you don't, you know, a feedback loop, you you you know, you want it to be correct, you want it to be making good changes, but how do you know it is actually making the right changes? So it's everything needs that feedback. Um super interesting.

30:47 Yeah, and that's why so measurement is extremely important. That's something I think we haven't touched on in this discussion, but measuring um the impact of of changes is actually really um very important it is because otherwise if you're just you know you're just making changes, but if you can't observe that uh that's having a good effect, then you can't really trust that anything is happening.

31:14 Yeah, and I was actually gonna ask the next question, which was around human interaction. So when do when do human like obviously with loops and and really the goal of this is to really increase that autonomy, increase that trust. When do humans get involved? And I guess I can kind of like follow on to your point there in terms of like that observing what good should be and and things like that.

31:37 When when does a human actually enter that that being part of that judging process?

31:45 Everywhere, Simon, and all the time. Um I think I think many people might find it a little bit daunting that we're moving towards so much automation, uh, but actually I think it's just uh an abstraction level. We are still doing so much work to uh like analyze what's happening, um request the right changes, monitor if the loops are performing well or not, coming up with new metrics that we think will um you know reflect better if something is working or not, all of these things.

32:23 It never ends. I don't think the factory is something that can just be built and done.

32:28 Yeah.

32:28 Like with any software, it it it's especially when you make the your code base, uh, your context and your data part of it. Uh it it constantly evolves and you're part of this evolution, and uh that is where the work is rather than code implementation.

32:45 Yeah. And and in terms of roles, let's get let's talk a little bit about roles for humans in the in the software development um world going forward with software factories. I guess software factories, if we have 10 repos, should we have 10 software factories, or are there shared components where you might have a shared orchestrator and and and multiple uh more specific components that are repo specific?

33:14 Um how much of this how much of this rule require humans to to manage versus actually building the product? Is there going to be is there a role where a human will do both of those as part of their job, do you think? Or do you think there's more of an infra role where people manage the software factory and more of a app development role where people manage the you know what needs to be built and how it gets built?

33:41 A really good question, Simon. I really like this.

33:43 These are all good questions, yeah.

33:47 Why am I surprised? Um at Tesla internally, we actually, um, especially when it comes to metrics and uh improving on those metrics, we actually talk about several different personas. Uh one of the personas we call uh an operator, dark factory operator. So I think this is what you just described as like an infrarole.

34:11 Um this is somebody who really cares about the uh main loops of the dark factory, um, and make sure the dark factory is operational and it can service everyone and so on. Um and then we have the steward role. And the steward role, we have many stewards at Tesla. Uh they're stewards of different areas or different projects, um, different repos, if you wish.

34:36 And it's kind of like uh the new code owner, if that makes sense. So the code owner is funny though, because they didn't write the code. That's the idea, right? That we like everything goes through the factory. Uh so that's why stewards. So they know about this area, they they are kind of responsible for this code base.

34:56 Uh and they make sure that the dark factory operates well in their code base. So there is um I think both roles need to think about the factory and think about this new way of development, but in a different through different lenses, if that makes sense. Yeah. Um whether this is one dark factory or many, I think I think both.

35:24 I think it's both one dark factory, and also each project, each repo is its own thing, but it can go through the common orchestrate uh orchestration, but then it it can define its own, it can and it should define its own feedback loops and its own special purpose loops and so on, um, so that it works better.

35:44 Yeah, I think that's really key as well, that you know, you can't just expect one repo to you know copy paste all the context and everything in terms of how to operate um and expect it to work just as well in the next one. It's everything needs to iterate and learn from the repo it's in, um, which is which is super important.

36:00 Yeah, absolutely.

36:01 Um Maria, what advice would you give people who haven't you know kicked off their first loop but really wants to actually build, wants to they like what they hear and they want to build a loop. What what advice would you give them?

36:15 Come to AIDEF CON Brilliant. Uh yeah, no, I think that might be very helpful. Um I would say what I was referring to before in terms of how do you how do you get going with one of those feedback loops, uh be very clear with your intent. Think about what's the data source, think about what um what signals, especially signals that when repeated make sense are in um can be found uh in that data source, and then think about what can you improve based on this.

36:56 In fact, and and perhaps focus on those pain points that you know what is it that your agents are tripping up on most, and almost like start building the loops around those because those are most likely the areas it can it can loop and improve and feedback the most.

37:10 Absolutely, because then it can improve its own behavior. Yeah, it's it becomes uh self-improvement.

37:16 Yeah. Amazing. Well, Maria, you you've been on the podcast before, you've looped right round and been on the podcast uh once again. Um thank you very, very much. Really amazing insights and appreciate you uh joining us today.

37:27 Thank you, Simon. Really appreciate it.

37:29 Thank you very much everyone for listening and uh see you soon on another episode. Bye for now. The AI Native Dev is brought to you by Tesla, the package manager for skills and context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community.

37:47 And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io forward slash community to learn more, and I hope to see you there.