Legora CTO: Why AI Agents Need More Than a Chat Window

0:00 Let's say you're getting sued and there's like multiple strategies and multiple ways that you could handle this case. That requires a lot of risk judgment, a lot of client collaboration, and I'm sure if you go to different partners at different law firms, they will probably all give you a different answer.

0:13 If there's no objective truth, it should be the human who makes that judgment. The question I ask there is just why? Why doesn't the AI just choose one? Let's say I'm a client and I'm trying to get this thing done. Trying to just go with whatever AI says. Yeah, you you probably could.

0:28 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

0:50 Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works.

1:14 This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. As a podcast listener, you can get 15% off your ticket using promo code Pod50.

1:39 We have so much planned, and we can't wait to have you join us in New York. Hello and welcome to another episode of the AI Native Dev. My name is Simon Maple. I'm your host for the day. And joining me is Jacob Lauritsen, who is the CTO of Ligora. Jacob, welcome to the episode. How are you? I am doing great.

2:02 Thanks for having me. That's a pleasure. And uh Jacob, tell us a little bit about uh yourself and also for those who haven't heard of Ligora, tell us a little bit about what you do.

2:11 So, yeah, my name's Jacob. I'm the CTO at Ligora. I've been with Ligora for about two years, a bit more. Um Ligora is a sort of workspace uh product platform for legal work. So we help lawyers basically do all of their stuff much faster, much better with AI. So it's sort of uh analogous to um in our world, uh GitHub linear uh cursor all in one.

2:38 That's Ligora for Laurence.

2:40 Amazing. And yeah, it like you know, a lot of people who maybe you know haven't haven't dived into the the the legal AI side of things may not may not be familiar. So it's great to it's great to hear that perspective. Um one thing for those who haven't heard is that uh Lagora have hit 100 million ARR across 18 months.

2:59 Uh is that is that around correct? I didn't misquote there. That's no, you're right.

3:04 We it took us 18 months to get to 100 million, and uh last week we just announced 200 million, and that took us six months.

3:10 Wow, congratulations. It's amazing when companies uh do this. What is it what does it look like from the perspective of the engine, the head of engineering, the the CTO?

3:20 Um I mean it's I think it's just always exciting that people get value from the product that we're building. Um that's sort of the the thing for us, right? And then it's it's you know, you can imagine a company scaling uh so quickly as we're scaling that it's a little bit chaotic. You know, it's a little bit a lot of things are happening, uh a lot of traffic, we have so much usage now.

3:41 Um but it's super exciting. I mean, everyone's super thrilled.

3:44 Amazing. And let's let's look a little bit under the covers because obviously that kind of like speed of growth from a company requires huge flexibility and speed from the point of view of the engineering team, the company underneath. Um, and would love to jump into some of the AI practices that you use.

4:00 Um from what I understand, AI writes over half of Logora's code base, and I hear it's I think uh all of it, almost. Oh, pretty much all of it. Wow.

4:11 All of it, yeah, yeah, yeah, yeah.

4:12 Now I heard some engineers uh can run 10 concurrent agents um at the same time. First of all, I'd love to understand how quickly did your team get to such a level? Um, because you know I've played with two or three, and it does take time to actually you know go beyond a couple or a few. And um and then secondly, I'd love to hear maybe some of the pain points along the way.

4:37 What were the some of the some of the problems or or dare I say outages maybe that that these kind of uh practices caused?

4:44 Yeah. Um first question how quickly did people get here? I that's very quickly. I think we've always hired for people that are extremely forward-leaning and people that challenge themselves all the time. And we have that type of uh talent pool, they'll adopt technology that I guess really quickly. We going from a few agents to a lot of agents was also relatively quick.

5:10 There was a lot of tooling. I think you're asked about pain points. So that was definitely one of them, is like um building the tools that let us orchestrate many agents. We have um we built a custom background sort of coding agent that we could launch. Um, we invested into uh cloud dev setups so that one agent can have its own cloud setup and run its own look host and observability and everything and send back videos and screenshots of what it's working on.

5:35 Uh, but also our own. I mean, I have there's a few engineers that have that are sharing this like uh Tui that they built, which is like a canben board for agents. And that's one way to orchestrate a bunch of agents. Um another one is we spent some time figuring out how we could have multiple work trees, uh work on concurrent things, but still sharing a lot of resources for local dev.

5:58 So, for example, you know, our app requires spinning up Postgres and Redis and all these types of different um dependencies and observability stack, et cetera. And then we built this custom tool that allows us to have one instance of all of those running and then sharing them across all of our workspaces so all agents have their own, you know, database in Postgres and their own prefix and Redis and whatever it is, so that we can use uh the resources on our local laptops efficiently.

6:23 Nice. And I and I presume the parallelization here is happening across not just a single feature or piece of work uh like you maybe you would on a work tree, but this is presumably happening. You're you're you're having individual developers running multiple pieces of functionality and and and running different tickets at the same time.

6:40 Yeah, absolutely. And that's just on the developers that are working on their features or their whatever they're building on now. But we have full loops that don't even involve developers or developers um machines themselves, right? We have uh our buck reporting Slack channel when there's a buck that comes in, it kicks off a cloud agent that tries to reproduce it.

6:58 And if it can reproduce it, it sends video evidence back into Slack and then it fixes it and writes a regression test and opens a PR. Uh and then an engineer, a goalie, will take a look and review and get that merged. So we have whole sort of loops and work that runs outside of any actually engineer.

7:15 Yeah. I'd love to also double down on one thing that you mentioned, which is you know the types of engineers that you that you hire, you know, they absolutely need to be that AI native, they need to be forward AI thinking. Um when we think about senior developers who are, you know, I guess what we would call old school senior developers versus uh a young, well maybe I shouldn't say young, a more an AI-first developer who hasn't been, let's say, tainted by the more traditional uh styles of software development.

7:48 Do you do you have an inkling to which you would almost like prefer or which you see the better results from?

7:55 I think the generally people that are very fast learners, that's strictly a good thing.

8:02 Yeah.

8:02 So people that are very curious and super ambitious to learn and try new things, and those that's the type of people we have for, regardless of age or background. Then there's um some, you know, there's definitely some advantages to being very senior and having seen a lot of stuff, especially when it comes to really big scale or distributed systems or databases, um, where it's just very helpful to have seen some stuff.

8:31 Um but then it's all about finding those people that have seen a lot of stuff and have a lot of you know experience, but are still extremely forward-leaning. The there's a Venn diagram and some overlap there, and then those pupils, I find if you have a lot of experience and you're a really good engineer and architect, and you're then also super AI pilled and really good at using agents, then you're like a force to be reckoned with.

8:51 I think that's like a those engineers are just extremely valuable.

8:55 Yeah. It seems to me sometimes you have you know a lot of people that are experienced but almost are held back from trying to do things the the uh a more traditional old way versus you know trying to trying to fix problems that may not even exist in today's world. And and I and I I even heard someone say, you know, that that they'll comfortably substitute some of that experience for someone who is hungry and and ready to essentially prioritize speed.

9:22 Um yeah, I think this is really it comes down to the the person, right? I think um both are super valuable. Um I think speed is extremely important solving being pragmatic, solving tomorrow's problems are extremely important. But I think um some part of systems thinking and thinking six months ahead is also really, really important.

9:39 If you're gonna look around corners, it's really important, especially as you scale the team. I think if you're a small team of 10 engineers, then just you the only thing that matters is tomorrow. Like you just have to fight for survival at that point. Yeah. Um but if you're a hundred engineers and everyone has that mindset where it's just only about tomorrow, then you're gonna end up with a code base that's completely unmanageable and you're gonna have a lot of incidents and you can't really um you can't scale that very well.

10:02 Where if you have a few more systems thinkers, you can actually build a system where you can be a lot of people running on it and you can get a lot of gains from the technical investments that you're making.

10:10 So so I guess also, Jacob, the you know, from the point of view of a of you know someone in a senior position for as part of an engineering um organization, it's really important to be able to know how uh how how much your organization has adopted AI. Um now you've criticized token maxing as a way of measuring that adoption of AI.

10:33 First of all, talk us through that a little bit.

10:37 Um yeah, I think there was probably an era, let's say maybe it was at as it at its peak six months ago, where everyone was talking about token maxing, and people had leaderboards of like how many tokens people are using, and leaders would track them and they would tell people you're not using enough tokens, and like some people would even build this into their career ladders, and they're like, You have to, you know, the more tokens you use, the better it is.

10:59 Um and then we just have I think we didn't, I so I didn't do this ever, but I I talked to other people in the industry and they um you know just have engineers that would just burn tokens to burn tokens, right? They just have them running in loops doing nothing. Um obviously that's not the right solution, and so I think uh I think it's always been a really bad way to encourage AI adoption by telling people that they need to use a lot of tokens.

11:23 And I think instead it's way more important to focus on um just setting a high expectation for output and what we can get done, and then you drive AI adoption through that as a means to get more amazing work done faster. Um, and you know, highlight all the wins and and you know, do some some hackathons where people show how you can work now and and open up people's eyes a little bit.

11:46 I think, yeah, that's how I think about it. And I think the the industry is also coming around that the token maxing doesn't make any sense. Um and it's just more about setting high expectations.

11:55 Yeah, it feels like the new lines of code uh uh you know quite. Yeah, exactly. That's exactly what it is. And and when I do hear people say, oh yeah, if you're not token maxing, you're not using the most out of AI, most of the time they come from a Frontier lab, and obviously it's in their best interest to say that.

12:12 So what's on your what's on your dashboard at Lagora to understand whether you are you know effectively adopting AI technologies?

12:23 I'm um let's see, I'm not the biggest proponent of trying to measure everything. The thing that's most important to me is are the teams delivering great stuff really quickly? That's basically at the end of the day, that's what I care about. Is the stuff that we're building great? Are the products great?

12:42 Is the architecture great? And is it, you know, we're doing it fast? Um and that's you know, honestly, if a team did it without using any AI at all, that's great. Then, you know, that I don't actually really care. Um, but I I don't think that's possible. I think you need to use AI, and I think AI is like an extremely powerful leverage way to get to that result.

13:00 Um so I mostly care about the result, and that's what I'm measuring on. Are, you know, are SLOs as lice great, are bugs low, that when I when I review the system with staff engineers and architects, etc., does it make sense? And are we delivering projects really fast? Um then we have a team, a developer experience team, a developer platform team, that's looking a lot more at AI usage to just make it very efficient.

13:21 And so I think we're looking at things like okay, how many turns does it take us to get to the place where we want to be? How uh how have we architected our uh system and our engine to produce code with agents in a way that's really fast and efficient and in a way that reflects how we want to build the system and how we want the system to evolve?

13:39 That's like a powerful way as a platform for all of engineers. Um that's something that I think a lot about is how can we invest in the right things as an organization such that all engineers are 5%, 10% more effective. That's like, I think for many years that's been what a lot of um leaders have thought about, but now it's actually really possible because you actually have this sort of one knob that you can can really invest into, which is AI uh enablement and sort of AI guardrails and whatever it is you set up, that actually has a really meaningful, very quick impact on um the engineering work that happens.

14:13 Yeah, and I guess there's like uh AI assistance, AI uh enablement. And when we talk about the autonomy of AI and the ability for it to actually replace tasks or replace pieces of uh pieces of work that would normally maybe you know we would find less interesting uh or time consuming. How what's your what's your belief on how uh how ready uh agents and AI models are in a to actually start replacing uh pieces of of of human work?

14:44 What what have you done in that space at Logora?

14:48 Um so I think if I look at this from our product and you know the illegal AI can do all of the menial work, all of the stuff of like looking through a million documents to find red flags and analyzing huge contracts against various laws and doing the legal research that's extremely time consuming and actually very difficult to do.

15:08 AI can do all of that now, and so it takes away all the menial, low-value tasks that lawyers used to be doing all the time, and they can focus on the high-value judgment strategic work, which is really why people became lawyers. You know, no one became a lawyer to sit and then look through a million huge documents.

15:25 No one's excited about that. Um whether and how we take that you know further, um, I absolutely believe we can do that, but there's some parts that will always require human judgment, um, or at least for the next 10 years, say. Um there's certain things where um so let's think about it in the way of verification, right?

15:48 Certain tasks can be, you can verify them, and that's typically the task that AI will be really good at. So that could be math, coding. Like you can you can get AI to do something and you can very quickly check if it's correct. You can if with coding, you can run tests. That's the way to make sort of code verification.

16:02 You see a lot of people now that are like optimizing towards a single measure. Because it's verifiable, a AI is really good at it. You can, you know, I'm sure that you've read a bunch of blog posts and lots of people doing certain things. I mean, the the um bun rewrite is a great example of this where there was a really big, really good test suite, which made me meant that we could just like do this huge thing because we could test it and then we'd know if it worked.

16:24 With legal, that's pretty different, difficult because a lot of the tasks aren't verifiable. And a lot of them also require this sort of strategic thinking and human judgment. And so some tasks, if it's a strategy in a litigate, let's say you're getting sued, and there's like multiple strategies and multiple ways that you could um handle this case, that requires a lot of risk judgment, a lot of client collaboration.

16:46 And I'm sure if you go to different partners at different law firms, they will probably all give you a different answer. And I think that's one of the tests where like if that's the case, that there's no objective truth, then AI won't be able to do it. It will still be able to do um to give you extreme amounts of value, but it won't, you know, it it can't do it autonomously, basically.

17:05 So there's there's there's a limit to certain things where you need human judgment. And then I think our job is how do we make an interface or a program where it's extremely efficient to sort of um bring your human judgment into the system and then agents can run at all of it. How do we make that interface with like all the work that happens down here?

17:23 Because it's so much work that all agents are doing, but then make it very easy to provide human judgment into the task where it needs to.

17:30 Super interesting. And I think you know, obviously, here from a legal case, the ramification of getting something right or wrong is is night and day. Um I I guess this brings me to actually to another very recent I say very recent, very recent at the time of recording uh a tool, which is you know a decision-making model like uh like Jev.

17:49 And and we've actually done a lot of playing with this uh ourselves, and specifically Yeah, I'm sure you have as well. So specifically in and around I mean like LMs, yeah, they could be great, they could be good at that kind of like decision making. However, a tool like Jev, where or a model like Jev rather, where it does provide you with you know an answer one way or another with a confidence, which is very, very cheap, very, very fast.

18:13 Um this feels like a no-brainer that kind of like replaces huge amounts of the, I guess the decision making, a lot of the workflow potentially, and then sharing back to uh to us as humans when confidence is you know below a certain threshold of confidence. Is is is this gonna really change the way we we model our agents, our applications and workflows?

18:35 Yeah, I think that's absolutely possible. I think it's um it's almost weird to me now that decision models, even the early ones, the Gliner, you know, whatever versions you had of this before wasn't much bigger. It's it, you know, looking back, it's so obvious that a lot of AI stuff today and a lot of application AI is decision making and not writing.

18:56 You know, you don't need an auto-regressive model to do decisions, and and we're using those everywhere, right, to make decisions. Um obviously when you need to do writing or drafting or whatever it is. Um, but no, it's definitely something that's very top of mind for me is how can we use this? How can we decompose certain of our tasks?

19:12 We're already really good at doing this, um, and introduce something like this in various parts. Um, we haven't yet found out if the confidence scores that you get from a decision model are trustworthy. I remember like I used to do these big classification models with like classifier heads and BERT models, and generally the the confidence uh scores weren't really trustworthy in any meaningful way.

19:34 So I don't know if I'll trust those scores too much. But um, hopefully, uh I haven't I haven't looked at Jeff specifically on on how much I trust those scores, but um, hopefully they're calibrated really well, and if they can be calibrated very well, or maybe we will fine-tune one for our domain that's well calibrated, then that that's certainly a very powerful um primitive to use uh across our product.

19:55 Yeah, and I was actually talking with someone today in our AI engineering team about something whereby obviously LLMs have been very much made for text and chat, but they also have structured output. And so we we were actually wondering, well, you know, if it has structured output, there is a world where we can actually change a model so that it does suggest actually provide very similar output in terms of you know a yes-no with a confidence level.

20:20 Um but I presume this is still gonna you know you know take time, it's a general model still and so forth. But it's it's it's very interesting. I haven't we haven't run any tests locally here to compare that. I'm not I'm not sure if if you have or if you have seen anything like that yet either.

20:34 No, well, not not any, not big enough that I feel comfortable sharing anything. But um we actually did have we have a product where we're actually doing this exact thing where we basically asked an LLM for this is like two years back and it's still working really well, where we actually asked it to basically do this.

20:48 Like, here's here's the answer, here's the confidence, uh here's how it works, and it's like you're just that's just a decision model, right?

20:56 So that's right in. It it's interesting because uh structured output is actually something that isn't new, it is a couple of years old, right? Like like you say. Um so human input, let's talk let's talk a little bit about when we get humans involved, the the the meat bags of of uh world today. Um at what stage should they be involved in decisions then?

21:15 We talked about low confidence, um, I presume critical decisions as well or critical projects. So, for example, a legal case where something you know really does need verification. How do you decide when autonomy can can really be left alone to the to the agents to decide and when humans need to be uh involved?

21:36 Um yeah, this is a great question. I'm not sure if we've yet nailed it. I don't know if we ever will, but it's um I mean there's obviously user preference, so we can't just make a universal truth. There's some users that are maybe new in their journey with AI and they want to confirm everything. Just like you know, us in Cloud Code initially when we had to say yes to every um sort of tool they wanted to run, we'd say yes, yes, yes.

21:58 And then now there's auto mode, which is I think all engineers basically just run it on all the auto mode, or maybe they even ran it on YOLO mode before. So I think we need we need to bring everyone along for the ride. And so there's there's some user preference there where it's like maybe you want to uh review everything initially, and then as you get more comfortable, um we can do more and more stuff autonomously.

22:18 And um I really am working towards this future where it's more the agent soliciting impact from the human uh than the other way around. And so I think we're moving towards having a lot more proactive agents that are always on and always doing work, and then they will come and they will say, hey, here I need some input.

22:36 You know, we found this uh red flag in this contract. How do you want us to handle it? There are these two paths, or whatever it might be. And I think that's a very effective, you know, speaking about an if an interface for effective human judgment input, I think that's that's an example of a really efficient way to provide human judgment.

22:53 Yeah, yeah. And I think it it's one of those things that is it's a constantly moving goalpost as well as models change and as as as we adjust as well, and and what our users expect from the product. It's a it's a tough one to think if you nail it, it might not be the same next week, even if you change it.

23:07 I think no, exactly.

23:08 It changes every six months, basically, right? You're just everyone feels more and more comfortable letting AI do more and more and more. And so, you know, maybe in two years no one's gonna talk about this because AI will just you know we'll just always let it rip.

23:18 That's it, that's it. And it's funny how every single time, like we talk about JEV being released, and all of a sudden you know, you get a new Luna, Luna 6 comes out, and it's actually it we in our tests it's a little bit slower than 5.6, but it's actually very it's half the price of 5.6. So it's actually you see these model you know vendors adjusting constantly on the flight.

23:39 It must be so stressful being a it is very excited about because I think in the past year it's been mostly focused on Frontier. Like the labs are very much like can we solve more millennium price problems? Can we like discover new drugs? It's very much which makes sense because they're trying to push AGI.

23:57 But I think for a lot of application companies like us and and you know loads of others that I talked to that are in simple solution, we're very excited about the entire period of Frontier. And we're very excited about a Luna that's faster and cheaper and the same level of intelligence. Like we can use that a lot.

24:10 Not all tasks in coding or in legal or in finance require, you know, a country of geniuses in a data center to solve it. Some of it is just requires completely ordinary intelligence, but if it's very fast and cheap, that's great.

24:23 And this is really interesting because I know at Liguri you benchmark 10 plus maybe models every single week. And I know you switch between the models pretty much. So uh are you using different models for different tasks? Yeah.

24:36 Yeah, yeah. We um and when we do this in in various different ways, we decompose tasks into different subagents and subcomponents. Um we have certain um different products that will use different models. There's various ways that we decompose tasks and we use different models. Um and yeah, we change them per task, you know, almost per week, uh, depending on what's best.

24:59 Because we're always um our basically part of our offering to our clients is that we uh give them burrito efficient AI. We give them easy to use, efficient AI, abstracted away from them. They get the best of what exists, and we are really experts at figuring that out. Um and to do to do that, we have to test these models all the time and always give them what's burrito efficient.

25:23 Um and that's also why people you know go to us and law firms trust us and corporates trust us because they know that um not only do they get great AI, which was the most important thing a year ago, but now it's great AI at a great cost, is equally important.

25:37 I I remember, gosh, years now, two years ago, uh we had Dez Trainer on who is the uh who is the founder of or co-founder of Intercom. And he talked about these things called torture tests, which you know are just a really ragged set of tests uh that that essentially run uh testing a model uh based on you know various different user requests and and things like that.

25:59 So they essentially replay a whole bunch of some of the trickiest uh user questions there are. When you're changing models constantly, do you find you know is that is that still a thing where you actually need a really rigid, or not rigid, but a really robust set of kind of like torture style tests to make sure you're not regressing in production or are are or are models you know not quite so varied these days?

26:22 No, no, it actually is. It's it's definitely something, but it's not just models. We need these both for models but also for all the harness stuff we do. I mean, we might change the way that we decompose the task, or we might change the way we do compaction, or we might change the way that we search in files or whatever it is.

26:34 We might change some prompts or like tool search, whatever it is we change, you know, we we we want to iterate super fast on the harness and not just on the model. And and so we need these really uh extensive eval sets to make sure that you know every time we get something new, a hard thing or something's not working, we add it to the eval set and as a regression eval or test.

26:51 And so we need to run these all the time, every time we make changes to make sure that the agent performs um at its best possible all the time.

26:58 Yeah, super interesting. Um I want to come back to something that you've said in the past, which is agents are not humans, and we should not constrain them to human language. Um I really like this, particularly particularly when we kind of, you know, when we think about the interactions and we talked about Jev that you know it is now designed to give us an output that isn't supposed to be, you know, in any way like a chat-based thing.

27:20 Um and this kind of like lends us leads us into a kind of like an AI native interface or or in a in a place whereby you know we we expect you know an agent to take in very different inputs and provide very different outputs maybe for consumption by an agent uh rather than a human. Absolutely. What is uh what does an AI-native interface look like when we think about models, or how should we think about it?

27:44 I think there's layers to this. I mean I'm pretty intrigued generally about the way that we're solving intelligence via natural language, and we're I mean, even in robotics, um a lot of robots are humanoids. Um, which I think is interesting, right? Because it's just we're trying to create artificial intelligence to look very much like human intelligence, uh, or even look like humans and act like humans.

28:11 And maybe that's the right direction, but it seems inefficient to me. It seems like a maybe a necessary first step before we can actually move to something that makes more sense. I think there's probably a physical build of a robot that's way more efficient than looking like we do. Um just like and the same thing for for language models and for um intelligence.

28:30 Um but that's sort of at the very low end of the level. I think sort of if you look at a product like ours, um, I think the point that I was getting at back then was that um if you're doing really complex work, the most efficient interface, you know, going back to what I said before, for you to review that work and understand that work and guide that work might not be a very long single chat thread.

28:51 Um, that is what you and I, that's how I would collaborate with you if we were working on something together, that would essentially be you, me talking to you, you talking to me, and it's a bunch of messages back and forth. But um that's because that's just a human constraint. This is the only way we have to communicate.

29:07 Um but people have communicated in other ways, right? They've created slideshows or whatever. Um and agents can do all this sort of stuff right off right off the bat. So I think when you do a really big due diligence analysis, you're analyzing millions of documents, uh, giving me the output of that huge analysis as like a long chat message is not super efficient.

29:27 And so we've built a product, which is like a big Excel essentially sheet that has all the all the files as as rows and data points as columns. And that's a very efficient interface for me to review the work because I can very quickly go around and see what's actually happened. Um, and so I think that's like that's an example of how we think about certain when we do really complex agent agentic work, what's like a good interface to interact with that?

29:51 And this is one example. There are more. It's a you know constantly moving target. I don't know what it ends up being the right solution, but I'm not convinced that everything's just going to be chat.

30:02 And I presume that extends to your context as well, in terms of you know, your context doesn't necessarily need to be human readable or in a form that's uh you know consumable by us. It it kind of makes me think, you know, the the black box is turning even more closed as well. And how do we how do we actually you know understand if it's getting the right data and we we you know we rely sometimes on verification or tests, but you know, it's getting harder.

30:25 No, yeah, I I think you're right, but I think that's also solvable in the same system. Like that that needs to be part of the system of interaction. That needs part of the system of interaction needs to be auditability and understanding, uh, especially in our domain. If you're you're working on an extremely important case and you're a lawyer and you get an output from the core, you need to be able to verify it.

30:46 Right? So we have right now we have like multi-hop citations, basically, which is like everything is citable. You click on it, you see that where it came from, and that might be from a different report that was also AI generated, but then you can see its sources down to the source documents or multiple source documents.

31:00 So you can easily follow the chain of reasoning. And I think that's super important. And no matter how we build our system, we need to ensure that it's auditable and understandable and reliable by um that humans you know know what's happened and and can say okay to this.

31:15 Yeah. And staying on context, actually, the way you talk about it there, where you know there is obviously so much data, and there's you know very strong links and citations between various documents and things like that. How how how complex a problem is context and keeping your context windows a reasonable size, making sure you have all the data?

31:33 Is this is this a significant pain when you have such a large volume of data to be referencing? Significant pain. I don't know. Everything you're thinking is a significant pain. Um is this the most significant of your significant pains?

31:51 Um I don't think it's the most significant, but it's definitely up there. I mean, again, the the goalpost keeps moving, right? Initially, people were using Lagora for very simple things, and now they're using Lagora for like multiple hours, like they have agents that run for multiple hours and do extremely complex things.

32:06 Um and then it this this type of stuff is important, and we need to figure out how to how to solve that, and we have different approaches and different methodologies to do that. Um, but yeah, it's it's a difficult thing to build an agent that can produce uh that can work for like the human equivalent of days.

32:22 That's that's a difficult thing to do, and very few people have solved that yet, and that's something that we're uh continuous to work towards, and and we're making really good progress, and I'm very excited to see you know what the future brings there.

32:32 Yeah, it's it's very interesting. And and uh you once said, I think talking about context rot, that the solution for context rot isn't to actually grow the context window bigger, which you know sounds like it might be a reasonable thing. Um have you been burned by this in by you know working through Lagora or is is there is there a better path you found?

32:53 You mean using models that have bigger context windows to solve the idea of like stale context, context uh growing too quickly? Yeah, yeah, yeah, yeah. Um this is also a great question. Will we just get models that have a hundred million token context window and then there's no problem at all? Um maybe it doesn't look like that's the direction that things are going.

33:14 Um I do think Yeah, it's been interesting. If you look back over the past four years, say, or three years or whatever, there there was a very big um divergence in what AI was. It was a lot of very different things and different models and could do different things. And you decomposed a lot of inbuilt systems.

33:34 And then like over the past years, it's it's converted a lot, and like it's just LLMs, and you know, there's like three different tiers, maybe, but they're basic basically all the same. And and I'm wondering if we're starting to see a little bit of a divergence again with things like decision models and auto-regressive models.

33:48 Um and if we are, then it opens up for so many new different types of systems to be built and solving really complex problems again, might be decomposition, decomposing them into having like decision models take some decisions and it's like a tree and then it does this other thing. Um or maybe decision models end up you know only being useful in a small sliver of places and the the LLM stuff keeps going and we keep growing context lengths and they just get better and better and better, and then all the problems are solved.

34:16 Um I don't know yet. I think that's like we we just have to see and we have to play, so play both and see what works best for us and where the industry is going.

34:26 Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

34:43 But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

35:01 Alright, back to the episode. I don't want to leave this conversation without talking about how we verify answers, how we verify the you know solutions or or or answers that are given to us. So we'll talk a little bit about the verifier rule, verifier problems, um, and then I and then I and then I want to talk about litigations briefly as well.

35:18 So we'll talk about the verifier rule. Um this argument really states that that that litigation strategy is basically unverifiable. So if you have five lawyers, you'll get five answers. Now that means I guess that you know there's this there's this ceiling in terms of how well or how good agents can ever be, because no matter how good the agents get, they're never gonna be able to arrive at an answer if five different lawyers can't get to that same answer anyway.

35:48 So, what does good look like here, and how do you verify against that?

35:54 Um so I should start by just clarifying verifier's rule basically states, and I mentioned this a little bit before, basically states um if a problem is verifiable, then AI will solve it over time and it will do well. And verifiable means um that it basically means a few different things, but generally it's like if there's an objective truth and uh it's very quick to verify.

36:19 It might be really take a long time to solve, but once you have a solution, it's very quick to verify it and it's very cheap to verify it. And so math is a great example of this, where it's very easy for an LLM to output something and you can just run the actual math and see if it's right or wrong. Um, and the same goes for code.

36:35 If you write tests first, then again there's it's very quick to and and generally this um rule has been applied to model training and like the actual capabilities of a model. But I think the same thing goes for just like AI systems generally. Like if you have an agent and maybe the model itself isn't great at the same math.

36:53 If you put it in a harness where you just said no, it's wrong, no, it's wrong, then it will converge to the answer. Like it will get there. Um so that's the verifier's rule. And the the way you're to your um question here, will AI ever get so good that you know it will give us an answer to the things where there's no objective truth.

37:13 Uh it will be, we can probably get an answer. It's just gonna be one of many uh answers. Um I think what's interesting to us there is that that's the point where you need human judgment um to us, right? That's a human, whoever is like wants to get this thing done, whoever orders this task to run, should have that judgment and not an AI.

37:34 If there's no objective truth, it should be a human who makes that judgment.

37:37 Um the question I ask there is just why? Because if you have five different humans, yeah, you might as well take an AI judgment for it, you mean very much like you know, it it it it depends which human you ask at that stage. If you ask five different humans that are all lawyers, you might get five different answers.

37:52 So why doesn't the AI just choose one? And and it's a tough, it's a tough reality, right?

37:58 It it is, and I think that's it's you make a good point. It's like, okay, if if I was, let's say I'm a client and I'm trying to get this thing done, trying to just go with whatever AI says, yeah, you pr you you probably could. I think the difficulty is the humans that are there that you have, that you that you know that you hire your law firms, um, they have all the context for your business and what's important to you.

38:21 They know you really well. They know the risks that you're willing to accept and not accept, they know where your company's going in five years and the previous issues you've had. Um, can you put all that data into a system and then have a AI do it? Probably eventually you will. But I think right now that doesn't come in automatically.

38:36 If you ask an agent, hey, you know, draft whatever strategy we should have for this litigation case, um, it won't have that context. And I think that's why you still need that human to be like of all these five options, or these different paths we can take, I know the market super well, I know the client really well.

38:52 This is what makes the most sense. Over time, might that change? Yeah, maybe.

38:56 Yeah, it's interesting. It's a it's an error in knowledge rather than an error in judgment here at this stage. It's that the it's that the agents don't have that that level of that level of knowledge that we actually kind of I guess take for granted. Super interesting. And I guess um when the agent does get it wrong or does pick something that is potentially you know not what a human would do, um what what does that then mean?

39:19 I guess in certain cases that could be that could be a lawsuit, right? Uh have there been any you know lawsuits that have been raised? Uh well I I mean it's interesting, right? Because it's you know who who's who's responsible?

39:36 There's a there's a philosophical meta question to this, and there's like the the lived uh reality today. Um the lived reality today is that AI is still a great tool, and we make it very easy to verify what it's doing. We don't guarantee that it's always the right the right way that a the best lawyer in the world would do it.

39:56 And so all lawyers still look at it. They're still just like you and me, I'm still responsible for my PR that I put up, even though I did it with Quad, uh or even though Cursor wrote all the code. Um I'm still responsible that this approves the system. And it's it's similar with with um lawyers that they um it's their output still.

40:15 We help them a lot, we guide them, but it's still their output. And they can very easily in our system verify that this is what they wanted, this is what makes sense, this like reasoning makes sense. Um and no, we haven't had any losses.

40:29 I'm glad. I'm glad. So why don't we why don't we wrap? This has been super, super interesting, and I think it's very, it's very cool that you know a lot of the time we we we talk about people who are who are you know building applications, those applications are perhaps used and embedded into other systems and things like that or used by a user directly.

40:46 But this is this is a really interesting kind of use case where it's you know ultimately uh um uh I don't want to say a dangerous space, but it's a space that can have vast implications. Yeah, absolutely, the consequences. So let's take it a little step further and say, okay, if you had to bet, um, what would you say is a genuinely unverifiable, very judgment heavy uh piece of legal work that today and agents would defer to a human to make a decision.

41:15 But what would you say is is is the first, you know, of that type that an agent will actually end up doing well? And one of the first pieces, what what would you say is is next on that journey?

41:29 I I actually think there are quite a lot of these where humans are still in the loop now that agents will be able to do it. One example is when you draft a contract, you might be asked to like draft a new contract for XYZ reason. Then typically lawyers will look at what's called precedent. What's what have we drafted before?

41:45 Um, and so they'll go and search basically, and they'll be like, okay, we similar clauses for similar types of clients in similar situations, in similar jurisdictions. It's like a multidimensional search, essentially. You have to find one that makes sense, and then you have to apply judgment in like, would I rather take this precedent for a client that's similar but in a different jurisdiction?

42:04 Probably not. Would I would I rather have a similar client in a similar size of a company or similar, like a smaller client, a smaller company in the same industry? Uh would I rather have one that's, you know, whether the client asks for the same things. There's it's like a multi-dimensional judgment which one is the right one, and maybe there's three that are right, and then from that you draft.

42:23 Um, that's an example where AI will be able to do this very well, because it can apply judgment in like a multidimensional case, or with very little steer um steering from humans. Uh that's just the first example that I think of, but uh, there are actually there are many of these.

42:36 Yeah. Yeah, very interesting. I like the way that you're not actually taking the pure data in there, it's actually taking historical data on how a human has interacted with it in the past, which is super interesting. When you previously talked about, you know, it depends on that company, their level of risk, and things like that.

42:52 Well, once a human's done that several times, you can essentially put that into the system exactly.

42:56 That's that's really exciting.

42:57 Yeah, very, very nice. Um Jacob, this has been this has been amazingly good fun chatting with you about this. I really, really appreciate it. Um where can where can folks who are interested in this kind of uh you know legal application, where can they go for uh for to learn about it more, to start playing with it, get more, get more information about it.

43:15 Go to our website, lagoor.com. Go to laguar.com slash careers if you're interested. We're looking for uh I mean we're into this a little bit, we're looking for um really curious, super ambitious engineers um in New York, in London, in Stockholm, and in Malmur, which is by Copenhagen. Um and we help relocate and we help with Visa.

43:34 So anywhere in the world, if you're a really ambitious, you're a really great engineer, you should totally uh apply.

43:40 Amazing. Well, it sounds like great fun to a great fun environment to work in. So it sounds sounds excellent. Amazing. Jacob, thank you so so much. Really appreciate all your insights. I appreciate it.

43:49 Thanks so much for having me.

43:51 Thank you very much. And I hope you enjoyed that. Uh, a really fun episode on the AI Native Dev. And uh tune in to the next episode soon. Bye for now. The AI Native Dev is brought to you by Tesla, the package manager for skills and context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler.

44:09 The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tesla offices in central London. Visit TESL.io forward slash community to learn more, and I hope to see you there.