Using Agents for Long-Horizon Financial Work

We're actually not a SaaS company. We are, um, we are the firm ourselves. Agents prompt people more than people prompt agents- Mm... and it is a proactive kind of, um, service that is running for them twenty-four seven. [on-hold music] We're here at the, uh, AI Engineer New York conference, twenty twenty-six. Um, and I'm here with Aman Singh, uh, from Arca.

Um, and, uh, Arca's, uh, recently raised sixty-four million for an AI-native wealth management firm. Uh, so first of all, Aman, can you tell us a bit about your role at Arca and, um, the financial services you provide and, um, how you're using AI agents to run them? Yeah, for sure. Um, thanks for having me.

Um, my role at Arca is I lead the AI platform team, so we're building the underlying sort of, um, harness and our capabilities to kind of run the business with AI, which is our ultimate goal. Um, in terms of the services that we provide to our clients, um, we-- This is a broad spectrum of financial services.

So that includes, um, sort of financial advice, financial planning for, um, for their assets, um, and sort of, um, uh, tax planning and insurance and estate planning as, as people get wealthier. Um, maybe you are, um, someone who just sold a business, for example. Mm-hmm. Um, and there's lots of complexities with all the different partners, et cetera, right? So, uh, we can help you think through that.

Um, we also have clients that are, you know, uh, really large technology executives that, uh- Mm... have, uh, are trying to figure out a real plan for retiring, retiring early, right? Mm-hmm. Um, and also sm-small families that, um, uh, that are trying to buy their first home, for example. Yeah. Um, yeah.

So the, um, the clients that you have so far, are they, um, do you have to kind of talk them through using agents and, and why you're using agents and tell them a little bit about the technology behind it? Yeah. So, um, the role of AI at Arca is not to replace the advisor. It is to replace all of the work around the advisor. Mm. So our clients don't really interface with AI directly. Mm.

All they experience is a level of service that has not been, been, um, been possible in this industry before- Mm... delivered through the empathy and care of a human advisor. Um, so we're building the machinery behind, um, every client- Mm... that's constantly thinking about, um, you know, um, what's, what is the market doing, and how does it impact Richard? Mm.

You know, um, what are, what, what else is going on in your life? Like, what are you-- what are the s-the, the textures of your life that impact your finances? Um, and, uh, we're able to be way, way more proactive, way more responsive, um, and just deliver a better, um, kind of, uh, level of client service using AI that does all of the, the back office stuff. Okay.

So let's get into the, some of the technical aspects of it. Wh-which models are you using to power your agents and, um, how do you decide which model is best for, for which action? Yeah. Um, so today we're using, uh, the Frontier, uh, models from, um, from, uh, the Frontier Labs, like Anthropic and, uh, and OpenAI. Mm. Um, Claude is kind of our primary, um, um, model, has been, but- Mm-hmm...

um, we've gone through several iterations of models over the years. Um, we used to be in Gemini way back when- Mm... um, and then OpenAI and then, and then Anthropic, and what's guiding all of our decisions are, um, our benchmarks that we have internally for, um, all of the various, um, capabilities of our general harness, uh, that it has to express in, in running our business.

Um, and so, uh, just recently, for example, um, despite the external benchmarks claiming, um, great things for Luna, it didn't really quite work out for us as, as well as we had expected. It was way too slow or, um... And it was costing a lot. Mm. And so, um, our internal benchmarks kind of guide our model decisions, and, uh, as a principle, we, we want to be, um, model-agnostic- Mm...

um, and, and sort of not, um, uh, you know, um, take a dependency on, on a, a certain provider or a certain harness from one of these labs and kind of keep that, that optionality open for us. Okay. So, uh, you just mentioned, uh, harnesses, so you're using different harnesses depending on which model you're using? No. We have, uh, we have our, our own- Okay... uh, kind of built internally that is mo-model-agnostic.

Okay. Um, so, um, you know, when, when your, when a financial advisor is doing something like a taxation report or something like that, how does the harness decide which model to use? Is that automated? Well, yes. So we have, um, certain tiers of models based on different complexities of, of tasks. Mm-hmm. So a task like assisting with a tax plan or doing a tax analysis would get our, like, heaviest tier of models. Right.

Inside of that tier, it is a little bit flexible, so you could use, um, so like an Opus tier model or a Sonnet tier model, um, or a, a, a heavy GPT model. Mm. Um, but, um, based on, uh, there's a lot of, like, background activity that our agents are participating in, right? That is using the, a lower tier of model- Mm... um, based on the complexity of the task.

So that's kind of how we, how we make decisions- Yeah... uh, on, on which model serves which task. Okay. So is it the, um, uh, the financial advisor that's actually making that decision while they're using the pro-product or- Not really. Mm. Um, they, they're, they're not aware of- Okay... uh, of that happening.

We've abstracted this away- Okay... uh, from them entirely. They kind of just interact with our product, um, as they would. Mm. Um, and those decisions are made in-internally, uh, by us. Okay. So, um, in your presentation at, at the conference earlier today, you talked about a tax bill, uh, example. Uh, I just wondered if you could maybe walk us through, uh, an example of adjusting a client's, uh, portfolio, investment portfolio, um, you know, from identifying a need to maybe rebalance the portfolio to executing trades.

Can you talk us through, you know, what, what parts of that process the agent does and, and what parts the human advisor does? Right. Um, so just like in the talk, this is a thing that will span ten days. Um, and so agents are ultimately, um, responsible for delivering that outcome. But there are, are, are certain parts that they do and certain parts that they do not. Mm.

So in this example, um, when, uh, when the client emails us, uh, with this whole tax bill, th-there's a, there's a piece of the analysis of like, where is this client's money allocated today, and what is the most optimal way, um, to liquidate some of that to, to produce a hundred thousand dollars? Mm. That is best suited for agents. Right To take on as an analysis task Yeah So that they do exactly that Do they have certain limits though? That they can't, you know, they can't execute a trade at- Correct...

X amount? Yeah. So the analysis part is fully automated, but, um, they're unable to actually execute, uh, any sort of money movement or liquidation automatically. Mm-hmm. We have a very deep integration with our custodian, uh, that actually gives us APIs to be able to do that. Mm. Um, but we have strict guardrails around human in the loop for things that would either move money or liquidate assets, or even directly communicate with our, with our end clients, uh- Mm...

because there's a level of service that, that, uh, service, service, and quality that we expect. Um, so all of the like internal analysis will be done by agents, but the, the... They will also pull in the right person to be able to execute the trade when needed. Mm. And they'll pr- give a, give the person a plan that they, they're able to review and kind of like adjust, um, and yeah, go from there. Yeah.

Yeah, so you mentioned that, um, uh, these processes tend to be, you know, 10 days or at least l- longer sort of, uh, running time than most sort of agent stuff these days. Mm. So, um, when an agent kind of resumes a task, maybe days later than they, you know, than they started it, how do you restore, how do you check that the plan still fits the client's, uh- Mm-hmm... you know, uh, needs?

Yeah. Um, so this is an interesting, interesting one to think about, is we have tons of these signals like I described in the talk. Mm. Um, some are, for example, you can imagine there's an email thread that is receiving responses, and there's chatter happening, right? Mm. You can imagine that, that any ad- ad- additional response to that goes to the same agent that's kind of like reading that, that email thread and reacting to it, right? Mm-hmm. Mm.

But a, a fresh signal of, like a, like a phone call that was made in the middle, um, that is a brand-new signal that isn't attached with any other ongoing process. Yes. And so there's like a fresh agent that wakes up to figure out what this means, right? Mm. And the way they, way they understand what is happening right now is our internal memory systems. Mm.

So we have, um, we have this, this persistence of, for every client, um, what is happening right now? What are we trying to service? Who owns what? Mm. What has happened so far? Mm. What are we tracking for next, right? What, what needs to happen immediately in the next couple of days? Mm. What are we tracking for the long term, right? Mm. Which individual is responsible for which action?

All of this is being tracked by our memory systems- Mm-hmm... that our agents contribute to. But also, there is a, a kind of really sort of dreaming process that reflects on the state of the world as it relates to Richard, the client- Mm-hmm... um, and is able to kind of like update these memory systems to, to persist long-horizon kinda tasks. Mm.

So that agent that received a new signal quickly ramps up on what, uh, the situation is with the client, and it'll know, "Okay, we're, we're actually executing a big tax bill." Um- Mm... and we'll be able to see, okay, my signal relates to that tax bill- Mm... so let me kind of contribute to the, to, to the state of that world. Mm.

Can you talk a little bit about the, the safety sort of aspects that you've maybe built into your product or harness to, to, to make sure obviously that, uh, no money is lost along the way? Yeah. Um, so all of our safety restrictions are, are fully deterministic, and they, they take the decision-making off the, the agent's hands.

So for example, um, certain tools let the, let the agent kind of execute the tool itself. So if it's like a simple record-keeping kind of tool- Mm... um, it will execute it. We also, with every time a tool is called that writes data, we have, um, a LLM-based verification running on, uh, on that action. Um, these like fast, uh, LLM calls that, that analyze every single write to our database.

Um- Mm... so that's another layer of verification we're running. And then, uh, we also ingest a lot of like external information, so there's emails coming in, documents coming in, phone calls being made- Mm... which are targets for prompt injection. So we run, um, sort of verifications at the, at, at the edges that detect presence of certain, um, you know, flavors of prompt injection. Mm.

And then anytime there is a, um, there is a case where, uh, data leaves our system, for example, in like a, a web search in, in the most obvious case, but also if we're sending a calendar invite to Richard, right? Mm. That's an external kind of, uh, data where we have both deterministic checks on presence of certain information that we know is PII- Mm...

uh, but also a consensus of small models that, that ensure that there isn't any kind of data exfil being attempted. Mm. Um, so we have tons of different, different kind of guardrails built into the harness, uh, with the principle that we wanna kind of contain them at the edges of capability- Right... uh, and not earlier than that. Yes.

And, and because you're in the financial, uh, you know, industry, are there certain kind of regulations that, uh, are required that you have to sort of keep to in terms of the technology that you're using, or is that, is it just more about outcomes and, and the way you sort of deal with clients? Yeah. The, the industry cares about outcomes, but, um, compliance cares about auditability. Right. Right? Um, and so, um, our, um, our system is, is, is such that auditability is by default. Mm-hmm. Um, we have every single action that, that an agent takes produces a change log of what changed- Mm... um, who changed it, from which part of the system, and when. Mm. And this applies not just to agents, by the way.

It applies to human actors as well. Mm. So if you change something in the UI that affects a client, uh, we'll log that action. So it lets us both get auditability by default, but also power some of our agentic capabilities, where the same thing that, that would tracks like a UI change- Mm... also could be used to, like, inform the agent that that happened. Right.

Um, so auditability has been a big part of our, um, our story, both for obviously compliance and regulation, uh, but also for our, um, a- agentic loop. Yes. In your, uh, presentation, uh, you had a nice line, "Agents prompt people more than people prompt agents." Um, so how do the agents decide who to involve and, um, you know- What sort of, how much of the context that they need to provide that person in order to do what they need to do? Yeah. Um, another int- interesting one.

So, um, this is similar to how humans work in, in, in, uh, in our industry- Mm... is there is a process that we're tracking over 10 days. There's multiple actors involved. The, the actor at step seven may not need to know about this just yet, right? They're pulled in at the right time. Mm. And so our agents with our kind of like long memory persistence plus the daily reflection on what is happening, um, reflect on where the, the state of the world is at the moment, and they're able to pull in the right person at that time- Mm...

when it is, when, when there's a signal that, that implies that, that person must be pulled in. Mm. And all of this is, um, work that we have done to operationalize the context of how this industry works, but not just that, but how does Arca want it to work, and what are our principles of, of how work happens in this industry.

Mm. Um, and that is another advantage of us being sort of a- a, um, a not just a SaaS company. We're actually not a SaaS company. We are, um, we are the firm ourselves. Mm. So we can standardize the way this work even happens- Mm... um, so that our agents can participate in standard ways that, um, that work must happen. Mm.

So, um, early in the conference, uh, Anthropic, uh, did a workshop about their managed agents', uh, infrastructure, and a lot of it, uh, even the orchestration is becoming more automated now. So do you foresee kind of using that type of functionality? 'Cause o- obviously it presents a danger in that the human is not necessarily always in the loop anymore.

So, uh, d- have you had any thoughts about using that technology to automate even more of the process? Um, I think we are very bullish on the models from, uh, from Anthropic and, and labs like OpenAI- Mm... um, but less so on, on taking a dependency on a product like managed agents. Mm. Um, because managed agents, one of their, their key things is how the models kind of persist and, and contribute memory- Mm...

as to what is happening in the world, um, and all the operational knowledge that they need to be, uh, need to use to, to be successful. Yes. Um, and so letting them own that part of the business is kind of like, um, a, a risk because, um, owning that is... Owning your intelligence is kind of, uh, gives you the ability to impact the business. Mm.

If you, if you hand off, uh, the intelligence to, to another provider, you lose some of that control that you need to kind of, like, build something that is unique and tailored to your business to such a deg- great degree, um, that it can actually impact your business. Mm. And just on Anthropic, they'll, they've also recently released, uh, financial services, so they're partnering with a whole bunch of different firms. Mm-hmm.

Do you think what they're doing is kind of competitive to what you're trying to do, or do you see them as different things? Yeah. I think, uh, I think Anthropic getting into our kind of industry is, is very validating for, for our industry. Uh, but the way I think about it is that it is very different to be an, um, a, a services firm that has access to AI tools- Mm... versus a firm that was built to be AI native. Mm-hmm.

The firm with AI tools requires their users to prompt the us- the, the agents kind of to, to produce work or to do work. Mm. Um, but here, like I said, in, at Arca, agents prompt people more than people prompt agents- Mm... and it is a proactive kind of, um, service that is running for them twenty-four/seven, and it's kind of like running the business in the most standard way that we, that we expect at the level of service that we, that we, uh, that we aspire to- Mm...

uh, which looks very different if you're, if you're, um, a, an AI, AI... Sorry, uh, a company with AI tools. Yes. Yes. Okay. Um, so a little bit more about on the security side. So can you explain, like, what, uh, in your processes runs inside a sandbox, and, um, how do you verify, like, a change to a financial plan, uh, you know, before anyone acts on it? Um... Mm. Yeah.

[clears throat] So, um, the s- the, the way we use sandboxes is that the, the sandbox is one component of our harness. Mm. Our agent isn't actually inside of the sandbox. Mm. It uses the sandbox to offload computation. Mm. Um, and, uh, we've, we've built systems to make it safe for code inside of that sandbox to access our data. Mm.

Um, so we kind of like have this thing which translates the code it wrote into a strict format that we expect, throw away the code, and you, um, kinda instill guardrails on that format before it reaches your back end. Right. Um, so we allow the agents kind of to use, uh, code to query our data, which means that they can, like, aggregate it, they can munge it, they can kind of like work with it as they, as they see fit- Mm...

to produce such a thing like a, like a financial plan. Um, inside of the same sandbox, they produce this, you know, beautiful artifact that we, we can share with clients that has like... it's a PDF of like- Mm... "Here's your financial plan, and here's how we're do- planning to kind of like approach your life financially." Um, and once that is produced is where we run our verifiers.

So, um, it's a certain type of artifact that is produced from the sandbox. Yep. The moment that it's produced, we run bunch of different, different verifications on it, um, some deterministic that check for presence of like, "Hey, did you just put like a raw account number in there? You should not do that." Mm. Mm. Um, so deterministically we can check that, um, but also, um, fast models like, um, like the, the smaller models or JEV- Mm...

um, to analyze for presence of things that might make this artifact non-compliant. Like for example, you really should not be like, um, promising returns, uh, in, in that artifact. Mm. So that's another thing that we will check before it is, um, it is kind of publishable. Mm. Um, and the last thing I'll say on that is that, um, doing this is actually, um, makes our kind of compliance, uh, better than if humans were doing it because AI is- tireless at, at, at finding even the smallest mention of a non-compliant, um, uh, piece of writing in there.

Yeah. Um, so yeah, we'll continue to invest in, in, in more of that and kind of like- Mm... constrain at the capability edges. Yes. Yeah, you, uh, mentioned Jev there. Um, so y- you've started to use decision models, uh, as part- Mm-hmm... of your process. Uh, uh, what other, um, products, other than Jev, are you sort of looking at? Yeah.

So what we've noticed is, um, when, when models are, um, working with financial data, right? When they're answering questions that require financial knowledge- Mm... they are-- they're very confident in, in, uh, in the knowledge that they have. So a small example is that, uh, models will very happily use the tax brackets from twenty twenty-five- Mm...

um, to, to come up with an analysis and be like, "All right, that is it." Um, but that is, that is obviously incorrect. And so, um, that is one of the ways in which we're using decision models in, in our harness is to, to check when the, the model is about to make a financial recommendation or, or anything, any action that requires financial rigor- Mm...

will detect and will nudge the model to kind of use our knowledge base of verified facts about the world that is true today. Mm. And/or use their kind of web search. Yes. So it kind of makes it, um... This is kind of a, a, a point about the, the harness and, and owning the harness that lets you do things like this that, that help your, um, your models be, be accurate where it matters.

And do you use, um, decision models in, for example, recovery if, like, a trade fails to execute? Is there a decision model involved there, or is there another path to, to get the agent back on track? No, not today. That is not a use case that we're exploring today. Okay. Um, so, um, you know, if, if a step, a trade does fail, for example, how does the agent recover at, at the moment in your process?

So, um, so at the moment, we are not doing autonomous trading with, with agents. Okay. It is very much a, um... The agents will propose that we, we make trades. I see. Um- Yep... and it is fully up to kind of our, our experts to figure out if that is the right trade to be making- Right... and to execute the trade. Okay.

So, uh, on the, on the same sort of, uh, line of that, um, your slides, um, say that every staff edit or cancellation comes with a why and come-- it becomes an eval. So, um, how does that feedback, uh, get turned back into, uh, the system as a measurable improvement? Yeah. Um, that is another one of the advantages of us, um, being the business and not a SaaS company- Mm-hmm...

um, is that we have, um, we have the ability to have to require them to give us feedback. Um, and yeah, the way that we loop the feedback back into improvements is something that you would, um, you would expect is all of that feedback gets piped into, um, a data store- Mm-hmm... that, um, that couples that feedback with the trace of the agent that produced it. Yeah.

Um, and so we're able to kind of like take that, take, take that trace, um, put it, put it next to the, the feedback, and every night, we run this process where we take-- collect traces from, uh, the real world- Mm... from our agents in the wild, and run, run a meta agent on top that suggests changes to our harness. Mm.

So maybe tools that were mis-misleading or, um, things that were, were errored, or there's missing knowledge in our knowledge base. Mm-hmm. Um, and they'll surface this information to us, and then we'll, we'll be able to kind of, uh, improve it. Um, today, we still have a, a human in, in, like a human engineer in the loop before we actually ship a prompt change. Yes.

Um, I've noticed that, uh, that agents will, will make really nasty prompt changes if you leave them, leave them to it. Um- Mm... every little fix becomes the whole prompt. Mm. Uh, and they're, they're unable to stay meta and kind of generalize, uh, the fix into, into a prompt that generalizes. Mm-hmm.

Um, so but yeah, we have, we have, uh, feedback loops from our product that we collect directly as first-class product features- Yep... to collect evals into, into improvements. Hmm. Okay. Um, and you also mentioned in your presentation about, uh, turning knowledge of how a firm works into agent context, which is actually something a lot of, uh, forward deployed engineers, uh, do- try to do is, is turn an organization's kind of internal knowledge into, uh, actionable things for agents.

Mm-hmm. Um, what does that look like, uh, for your company technically, and, and how do you sort of keep it up to date? Yeah. So we have, um, a knowledge base of, um, of files that rep-- is represented as, as, um, as files. And, um, the way we keep that up to date is we- we've handed it to our experts to curate and update. Mm.

So it is a, a first-class, um, feature in our product. Uh, so on the left-hand side, there's a tab called Knowledge- Mm... which has the full, uh, operational context that our agents are using. Um, and as they, they work on, um, uh, work on things, uh, they're able to cite knowledge, and so it kind of like creates this, this loop of like, "Oh, we noticed that the knowledge is incorrect."

Um, and, um, we-- that gets surfaced to us in our trace analyses. Mm-hmm. And because it is a, a part of the product itself, uh, people are able to go in and, like, edit the, edit the scale, um, uh, directly. Okay. Um, that's how it kind of works. All right. Okay. So, um, you know, you mentioned you're, um, uh, uh, you're actually offering these services to, to clients through, you know, uh, I presume you have a staff of, um, financial advisors in your firm.

So, um, can you maybe give us an example of, uh, uh, some, you know, a client that you've had that's had a measurable improvement in their investments or taxation or whatever through the agent system and, and maybe some sort of details about what in the agent architecture kind of helped that client, you know, get such a good return?

Yeah. I'll give you a recent example. Um, this is, um, this person wasn't a client yet. Mm. Um, but they were, they were at the, the point of becoming a client. So we had all of their, uh, their documents, their statements, their current investments, et cetera. Mm. And this person was already working with an advisor. Um, and so, um, they meet with us.

They tell us that, "Hey, I have this advisor already. They're charging me fifteen hundred dollars, um, uh, for, for their services." Mm. Um, "And, uh, they're not very responsive to me, so I'm, I'm chatting with you." And so, um, our agents have kind of like, um, you know, read through all of those documents. They've analyzed them. They've analyzed what the person's goals are that they state, right? Yep.

Um, they've also analyzed the statement of, like, who this current advisor is, uh, right? And how much they're, they're charging them. And, um, they discover that, that there is-- there are fees that this person was charging them that they weren't aware of, uh, and surfaces it, it to them, um, and kind of like makes them not only aware of, um, of the fact that they're being overcharged, but also aware that that person wasn't even fiduciary.

Mm. Um, so they're not fu-- um, obligated to act in their interests. Mm. So it is-- this is the sort of, like, attention to detail that our agents bring about in, in-- when they're analyzing a financial picture- Mm... um, that, that sort of, um, would not have been possi-possible if we didn't have, um, have these.

There's several examples of, um, proactivity, uh, that our agents kind of like produce of, like, someone has a deadline coming up. Mm. Or they're at risk of missing their RMD or, um, that kind of thing. And so, um, this is difficult at scale for humans to keep track of, but it is the, the default behavior of, uh, of our agents at, at Orka. Yeah. Wonderful. Well, that's probably a good, uh, uh, point to stop it.

So thank you very much, Iman, for joining us- Thank you for having me... on Out in Space.

Using Agents for Long-Horizon Financial Work — Aman Singh, Arca

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin