OpenRouter: from Seed to Stripe — with OpenRouter’s Alex Atallah & AMP’s Anjney Midha

[on hold music] Okay, we are here in Anj's house. [laughs] Which is where all great startups in San Francisco start. Howdy. [laughs] Uh, and, uh, congrats on Cursor, Mistral. Uh, I don't-- God knows what else. You've got so much stuff going on. [laughs] There's, there's a lot going on. Well- Uh-... OpenRouter is probably the most, has been the most, I would say, like, one I'm excited about recently. Yeah, yeah.

And we have Alex, uh, first time on the pod, but, uh- Thanks for having me... you've been to IE a few times. I appreciate every time you've shown up, uh, for the community. Congrats. I just-- Like, what a journey. When I was looking back at your past posts, one of the earliest principles that I saw you write as a sort of product person is Pub/Sub as a product principle. Mm-hmm.

And I wanted to you to maybe explain how you think about what should exist in the, in the world. Yeah, the, the Pub/Sub piece, which was early twenty twenty-three, I didn't think about it until we, we, we talked, like, ten minutes ago, is about how there, there is, like, a way of thinking about products as an intersection between subscribing to data and publishing data.

And marketplaces are an easy, easy, easy example of this. You have suppliers that are publishing some kind of product to a SKU, and the SKU is kind of like a Pub/Sub topic that a consumer is subscribing to and just going to, like, consume whenever they want. And humans consume in a very, like, discreet, ad hoc way. It's not very scalable. You know, all their attention is on the topic when they're buying the thing, and their attention is nowhere else when that happens.

Um, agents and, and consumers of inference- Mm... don't act like that. They're consuming continuously, and they're changing the SKUs that they consume from all the time. So OpenRouter is sort of like a blend between a, you know, a normal API experience and a marketplace where we create model slug, we have the auto router, we have all kinds of, like, product SKUs that you can subscribe to, and, and then you can, like, continuously add, like, uh, derive value and make decisions based on those, uh, those consumers.

Yeah. This is something that was more consensus now, but not consensus when you guys started, which was that there is such a demand for swapping models and cha-changing things out and, uh, that people would not use the, use the native SDKs. I guess, uh, for each of you, what was your sort of realization moment that this would, this would be it? I, I... You've, you've given a talk at EIE about Alpaca as, like- Yeah, yeah...

one of, one of your inspiring moments. Alpaca, I can, like, rehash the Alpaca moment for a sec. Like, the very beginning, at the end of twenty twenty-two, OpenAI was the only game in town. There was, like, OpenAI, Cohere, um- Yes... and then a, a smattering of, of, like, early attempts at open-weight models. Yeah.

Um, when Llama came out in January of twenty twenty-three, it was like, "Whoa, really exciting. This is really big. It outperforms GPT-3 on, uh, one or two benchmarks." Uh, but you can't chat with it. It wasn't like an... It wasn't actually an engaging model, but it seemed like someone just needed to fix a couple things and do some RLHF on it to get it all the way there.

And Alpaca was the first model that I saw that did that. It only took six hundred dollars to do. A, a team at Stanford generated a bunch of synthetic data, fine-tuned Llama, and made Alpaca seven billion parameter model. Or was it, maybe it was thirteen b-billion parameters. And it was so good. Like, I was just, like, on an airplane using it. I, I... You know, in many cases, I, like...

you could not discern a ChatGPT versus an Alpaca result, and I figured if, if it was this easy to make a model, um, one, we have a whole new way of monetizing data for the first time. Um, you can just, like, take really valuable data and turn it into a service in six hundred dollars, uh, and that cost will probably go down over time. Two- When, when you say-- So sorry.

Uh, when you say monetizing your data as, uh, what eventually will become an MCP endpoint or, or as a- So it was a very abs-... as a training data for a model. Yeah, training data for a model. Awesome. Like, an abstract way of saying like, "Hey, I have this data." Compress it into a model. Like, it makes sense for me in my product, but, like, I could repackage it in the form of a model and sell it.

And so it's just a whole new business model for the economy. It also, of course, provides like, you know, a way of following what frontier labs are doing, but in a way that, like, a single developer or a small team of developers can roll on their own. And so that... Whe-whenever you have an example of that, like a breakout app that's doing really well, and then some kind of framework for imitating it with your, in your own flavor, you have an immediate ecosystem of like...

Like, an, an immediate ecosystem, like, should arise because there's just a huge gap between the, like, decisions that the single company is making and all of the variations in those decisions that, like, a wider ecosystem can, can create themselves. And so then, you know, you need a marketplace to, like, discover all of those, uh, services and all of those products.

There wasn't any place on the internet that, like, was like a home base for LLMs in terms of seeing how much they were being used and seeing who was using them and why. The, the closest would be Hugging Face. Hugging Face was the closest at the time. Yeah. Which they just started Hugging, like, a few years ago, before, before that. Yeah, and Hugging Face also didn't have the closed source models. Yeah.

Um, and they didn't, uh... you couldn't use the models at the time, um, and we-- there wasn't data about who was using that. There, there are, there are, like, a bunch of differences between OpenRouter and Hugging Face, and those differences felt really critical to me, es-especially when I was just trying to learn about LLMs and, like, why people are choosing, like, these...

different little ones that are emerging over time. Got it. And then Ansh, no stranger to wanting more model diversity, uh, at the time, uh, you know, you're a couple of years into your Anthropic journey, which we have covered in a previous podcast as well. What was your introduction to Alex? Well, the introduction was, I think, 13 years before that. Oh. Um, but the OpenRouter handshake actually happened right over there, if you remember. Yeah.

[laughs] Um, which was-- So Alex and I, uh, met, I, I believe it was sophomores now, if I remember correctly, at a Stanford Review, um- That's right... meeting for the first time. I think so, yeah. Yeah. Yeah. So Stanford Review was the libertarian newspaper on campus at Stanford that Peter Thiel had started back in the day.

And, uh, for whatever, for whatever reason, I, you know, Alex and I both showed up to one of the meetings. And I remember, um, the editor-in-chief was a mutual friend of ours. Lisa was really a really great editor-in-chief. You know, part, part of an editor-in-chief's job is to assign responsibilities to people and make sure the work gets done. Um, and I, I, I may be misremembering the details, but I, I remember wanting to-- It was kind of surprising to me that at the time there was no dedicated technology section in the newspaper.

[laughs] Um, you know- Because it's political, right? It is primarily- Like it's talking about- It originally started as like a poli- political-... rights and- Mm-hmm. Yeah. States and all those things. Correct. Yeah. But it, it-- to, to take us back in time, you may remember this, but, um, there was, there was this technology, uh, kind of re- legislation that was being debated called, uh, the Net Neutrality Act.

And net neutrality is like inherently this political concept, right? It's, it's about the regulation of com- internet broadband access. And so there was a community of us who were kind of technologists but also debating the politics of the technology, and I thought the Review would be a great place for-- to, like, write about that.

And I was working on, I, I think a net neutrality article, and I remember proposing, "Well, maybe you should start a technology kind of section." And Alex was one of the only pe-people who said, "Yes, that would be cool," and said-- I, I forget whether we ended up writing stuff together, but they-- that's when we first met, um, was 2011 or '12.

I forget which year it was. It was one of those. Um- Yeah. It was at Old Union, if I remember correctly. [laughs] That's where we used to meet. But, you know, along the way, Alex and I have had a chance to, to, to hang out often. And probably the, the, the time when we had the most professional overlap was when I was running the platform at Discord. Um, and it had become this explosive kind of platform for crypto- Yeah...

uh, and NFTs in the middle of the pandemic. Which also, by the way, you were in charge of safety and security as well, right? I was the head of platform, which meant all of the crypto sec- the DAO and NFT launch security debugging fell on me. There was a- And the phishing and- The phishing, the s- the, the social engineering attacks, like a ton of DDoS that we were getting hit by.

Um, that's around the time I star- started teaching security at sc- at Stanford, CS 153. And Alex was in the, uh, at OpenSea at the time, and I was trying to figure out how we, we could defend against all these attacks that we were... Like, and at, at, at peak, I forget if you remember how much NFT volume was running through- Discord... Discord.

But it was like a meaningful amount of like se-- It was like several billion dollars in NFT volume of GMV, so to speak, were running through the platform, and it was all coming from OpenSea. It was these, like buy, sell, trade- I mean, the D-... servers... the D in DAO is Discord. [laughing] Yes. And so that's when I think we had hung out professionally.

But a year after that, OpenAI gave Discord early access to GPT... Sorry, GPT-3. No, it was GPT 3.5 actually. Yeah. Yeah, GPT 3.5, which, which is the RL version of GPT-3. And that's around the time w- we, we made a Discord bot with, um, OpenAI for internal deployment, and that's when I, I realized we would need-- Like since I, since I was part of the deployment team- What was the use case?

There were two that were-- And there's, there's actually a post now called Discord is your place for AI with friends that somebody sent me recently that I wrote, um, and published in 2023. But there were two use cases. One was Clyde, which was the inter-- like a first-party friend inside of Discord that could help you set up your Discord server and talk to you about onboarding and get your friends to hang out more.

Um, and then there was content moderation. And one of the realizations we had with content moderation was the-- it would refuse to moderate. Like, it would just refuse our prompts because the RL- Mm-hmm... the post-training was-- We were very early in the post-training era, and it would just-- our prompts would trigger it, uh, its, like, guardrails.

And we told OpenAI, "Hey, guys, we need access to the weights because if we're gonna be doing content moderation at scale," we had two hundred and fifty million monthly active users, "we need more reliability that the model will do what we need it to." And they said, "Well, sorry, guys, that's not how this works. We're a closed source company."

And so that was my first realization that we needed open models, and the enterprises would need more control o- over capabilities, and then ultimately would need some kind of control plane or management system to orchestrate these open models. But there weren't no good op-- There, there were no good open alternatives until maybe six months later when Llama came out.

And six months after that, I led the series into Mistral, which was started by Guillaume and the Llama team. And they-- At that-- At around that time is when I remember hearing about Alex launching OpenRouter and going, "These worlds are gonna collide, and I don't know when it'll make sense to team up." But Alex was so early and could see...

I think he was totally right about this ecosystem starting with Llama that then needed like a, an easy layer to, to manage for. Especially for-- I, I was approaching it from the enterprise perspective 'cause I'd been that, like the-- As the VP of platform at Discord, it was my job to ensure w- that when we deployed models to, like, two hundred and fifty million users, they did what we wanted them to.

And that was very hard, um, 'cause if you outsourced it to the labs and they controlled the, the guardrails, and their guardrails are their safety policies Forbid the model from responding to your prompts, that was q-quite catastrophic. Yeah. But well, you know- Mm-hmm... like, a moderation is a thing that they wanna support, and obviously, beyond that, they would work-- OpenAI would work with you, uh, you know, presumably to give you a moderation endpoint- [chuckles]...

which they offer for free. It was an interesting use case, um, that they-- So they did give us a moderation endpoint. However, as you guys know, every Discord server is like a mini deployment of itself. And so the use case was, instead of having human moderators that have to interpret the norms of the community, you just give the-- They o-often, like every, you know, subreddit, Discord servers, public ones, have their own rules that the user, the- Oh, yeah...

the user is creating. They run the namespace- Right... Discord and, yeah. And then humans used to read those norms and then enforce it every day manually, like observing each message in these communities, and these communities have, like, millions of users. So we had a five thousand plus person team globally in the-- on the Discord content m-moderation team. These are outsourced contractors who had a really tough job.

And so the idea was instead, if you could give the norms of that server to the LLM, then the LLM would do custom moderation for that server. It's almost like a, like, in context moderation for that server. And many of those servers' norms just violated OpenAI's rules. And so that-- It, it was like we had our own custom eval, so each, like, each server had its own custom eval.

But Discord-- At the time, OpenAI's evals, we were all so primitive in our thinking about how to deploy these LLMs that often the, um, the post-training prompts were super heavy-handed. It said, "Oh, anything about Harry Potter, anything that has trademarked content, you know, don't-- refuse." And if it was a, if it was a fan co-- Harry Potter fan community, this was a real use case, that had a content moderation, the LLM would just refuse. Yeah. And that was just not precise enough.

[laughs] Another one that, that we heard was, like, if someone was trying to write, like, a detective story, and there's one chapter with a lot of violence, like maybe someone, like- Right... kills someone, the LLMs would just refuse to, like, help with that part of the story. Yeah. And then they-- like, the user would be like, "Okay, this, this is not, like, structurally inherent to LLMs.

There must be, like, some choice out there so that I can, like, switch to another model, um, when I'm getting, like, a refusal or a bad result from the, the main one that I have." And, and that, like, tension also drove need for a marketplace. Yeah. I think that is well, uh, uh, accepted now. What was it like back then when you were raising or, you know, starting this, um, did people get it?

Um, you know, w-what was the, some of the struggles? Uh, basically I like, I like getting stories out of him about how other VCs don't get it. [laughing] [laughs] So, like, anything, anything you wanna, you wanna s- uh, talk about no- you know, now that-- let's, let's call it the, the, you know, the, the early journey of OpenRouter is done, right?

You can obviously talk about some of the early day stuff. Uh, well, I was gonna say that, like, the, the biggest objection we got is, is big model win, which is- [laughs] Yes... ev- all the value- Scaling laws. Huh? Scaling laws. Yeah, scaling laws, um, and natural network effects are just gonna kind of accrue to one company, which will be like a-- it'll be a Google-style monopoly, just like how Google won the search market, um, by a, a large, large margin, um, and you'll just be fighting for scraps at the end, basically.

That was probably the biggest objection we got, you know. It is interesting that Google won the, the, the search engine race with such a huge margin. Um, you know, I think, like, had there been more interesting benchmarks or had, like, search engines been, you know, a bit m-- you know, had people, like, seen them a little bit more like LLMs, where they're services that you can build companies on top of, that might not have been the case.

Um, but LLMs don't merely have a user interface, they're also, like, ways of building entirely new businesses, and, you know, a Google-level monopoly would be like the Dutch East India Company times, you know, quadrillion in magnitude 'cause the whole economy ends up, like, depending on the one monopoly as well.

So it didn't seem like, you know, a g-- like, would be a really crazy outcome if that happened, and it's also less likely because the, the economics of, like, creating good competitors are, are much, m-like, much more decentralizable. Everything Alex said is true, and I came at it from a completely different perspective, which is- Yes, this is why we're here...

um, the scaling laws were never like-- in my mind, were always a feature, not a bug for why OpenRouter would be very valuable. Because, you know, I was one of the first investors in Anthropic, and it was obvious to me that other researchers in our friends group, and I went to grad school for machine learning, and I just had a lot of friends in the ML community who it was very o-obvious to us that the bitter lesson holds.

And so I was like, "Oh, like fantastic. Now we have at least two pro-proof points that compute scaling works." It was OpenAI and Anthropic. Um, and by the time I think we decided to team up on OpenRouter, I'd already invested in Mistral and Black Forest Labs and Luma. So there was multiple model companies and teams that I was, uh, working with.

But you did other modalities, whereas this is literally- Across different modalities... text. [laughs] Exactly. And it was so obvious to me that an ecosystem of different kinds of models were being created, um, and that this whole dom-na-narrative of, like, only one company will dominate like Google was, um, well, like, maybe true, but one, I don't believe that, but two, there was so much extraordinary innovation happening across several different research teams.

But the shared problem I was noticing across all of them was ou-often, you know, the research teams were fantastic at figuring out how to reason about new capabilities. They think in terms of capabilities, but never on, like, are not developer mindset oriented. Like, what happens after the training is done and the checkpoint comes out?

Like, you'd be shocked how, how, like, similar the early pre-training teams at OpenAI, uh, sorry, Anthropic, BFL, Mistral, uh, were in, in, in their, like, default approach to- Taking their imp- research out of the la, you know, lab and kind of scaling their impact, which is often, "Oh, the checkpoint is done.

Put it out as an API. Done." And then there'd be crickets. Um, in, in the case of Claude, the first Claude checkpoint was actually done a year before they released it internally. And then ChatGPT came out, and we decided, okay, yes, it's a good idea to, to release a Claude version externally, and they had no plan. Like, no, no, no plan for how to get developers to actually try it out.

And so if you go to the Claude one bl- blog post, you'll notice there are, like, three kind of developer examples for users of the API, and one is a Discord bot, and the second is Vivian, my wife's startup called Juni Learning. 'Cause... And then there was, like, Notion, um, because these were all friends of, like, the Anthropic team- [laughs]...

'cause that's how har- like, last minute the planning was around, "Hey, once the model's done training, how do you get it out to the world?" There was no distribution platform that understood what developers needed. All the, uh, the key management provisioning, like, simple, like, you know, endpoint management, versioning control, like, all these things that the scientists and researchers go, "I mean, that's plumbing.

I don't really think about it." Implementation detail. Right. And instead, Alex came at it from that perspective. And so, you know, it was so obvious to me that, like, every single lab I was funding would, would spend extr- like, literally sometimes billions of dollars into training, and then a checkpoint would be done, and there'd be crickets, like, during early access 'cause they're like, "Oh, that's right."

Like- [laughs]... it's hard to use a checkpoint to make anything. You actually need a whole bunch of plumbing around it to make it usable by a developer. And so by the time-- I think we... It was so obvious to me that a distribution platform like OpenRouter was critical to have in the ecosystem if we wanted there to be competition to Google. Like, unless...

You know, with Google, DeepMind is done training a new checkpoint, and then they push a button, and it gets blasted out across all their surfaces from Google Docs to, you know, um- Everywhere, even if I don't want it. Everywhere. [laughs] You wanna know about, like, on Android, like o- like overnight, they can deploy a new checkpoint to, like, a billion devices, right?

And that invisible infra advantage, distribution advantage, most people don't realize, but until OpenRouter showed up, it-- you had to think about all of that yourself as a model lab, and it was very daunting. You know, at Anthropic, I think it took more, more than 12 months to get to our first 10 million in revenue.

And in contrast with, with Black Forest Labs, I remember the early days, you, you guys had a conversation with the BFL team, and, uh, it, it was so simple for OpenRouter to say, "Oh, no problem. Like, the day you launch, we can send a million developers to you." You know, that, that was crazy. That was like a step function change in, in, like, power. Is that a real number, a million? I, I, I- Okay. All right.

I think today it's like 4 million. How many developers are on OpenRouter today? We-- Over 10, but- Yeah. Over 10 million, but, um, but, like, it's ha- it's hard to, you know- I, I don't know how- We do a lot of, like, you know, account deduping work, but yeah. No one- If you could get 1,000 developers, just to put it in context, if you get 1,000 developers who actually try the model on day one after you release it and just, like, do inference and give you feedback, that's 1,000 more developers than they knew how to get to on their own.

[laughs] Well, you know, BFL had a reputation, but yes. Uh- They had one with Stable Diffusion. Yeah. And with Mistral, uh, I don't know if you guys remember, but the first checkpoint they, they released was, like, torrents. It was, like, torrent weights. Yeah, they just put up a magnet link. There, there was no API.

Yeah. 'Cause they didn't-- they weren't infra people. [laughs] You know, like, it's like, okay, download these weights, and you guys go figure out how to host it. He has a story on his side, yeah. Yeah, I mean, in addition to the, like, building a really good developer experience around it, the marketing that we do, uh, like, for different models is totally different and perceived- Right...

totally differently- Right... from the marketing that a model lab does for itself. Yes, 1,000%. We are like a n- you know, neutral layer looking at this market like it's a big, dark room with all the corners completely obscure to users, and users are walking into the room and, like, feeling around- Yep...

and trying to figure out what objects to grab off the tables and, like, build into, uh, co- their companies. And it's an insane way of working. Like, uh, models are not products where you can just enumerate all their features onto a webpage. They're all black boxes, including the open weight ones. So you need to, like, shine lights on all corners of this room, um, so that people can see what makes this model good, and you need the company shining that light to be a neutral third party, which is what we specialize in.

So the, like, it's-- in addition to developer experience, there's also, like, a very important, like, marketing and product packaging component- Yeah... and a way of, like, um, routing and, and discovering models becomes, like, critical to your go-to-market as a provider or a model lab or a server tool and, and more in the future.

And, and this value, to your earlier point about how many VCs, like, you know, just don't... O-one of my biggest frustrations is that venture capitalists, many of them, like, just don't have any operating experience in the field. You know, so unlike a traditional investor who's just maybe come up through the ranks as, like, a associate working on financial modeling or maybe hasn't been a real operator in the field for, like, more than 10 years, which is a big part of the industry now, I had just arrived at a16z, like, a year after running the platform.

And so I knew what the challenges were of, like, building a real dev-- a great developer experience and actually, like, being able to create a working piece of software with, with a model. And there were a few, I won't name names, but there were investors who were looking at OpenRouter, um, and, you know, felt at the time, like, w-when I would compare notes with people, that it was just, I quote unquote, "Just a marketplace."

Yeah, just a thin layer, just a proxy- Correct... just a- A wrapper or whatever on other people's APIs. And I was like, "You have no idea how strategic the value that OpenRouter has created by being able to orchestrate even three- APIs in production. The amount of both engineering work and community design that goes into getting that actually live and running in production at the scale the OpenRouter team had started, this doesn't happen by default.

And then that was one of the things that stood out to me about Alex from the earliest days, like he just understood like these-- from a systems perspective, like how do you get these flywheels going? Like that stood out to me with OpenSea when we were working together on the NFT integration at Discord. Like Alex had a level of commun-- like systems thinking on how you get these flywheels going that m-most scientists and machine learning people just don't think, think of. Like we often think in terms of pre-training, mid-training, post-training- It's a linear stage... it's this linear pipeline.

There's no loop, yeah. Yeah. It wasn't until much later that the modern context feedback loop cycle really got standardized in the industry, but at the time, if you remember, machine learning was like-- like mostly we did a lot of ML like when I was in grad school on a laptop. So you just like download a dataset, ran some ablations, and you looked at the loss curves, and you're like, "Great, I made AI." And the idea that you have to like deploy those capabilities, collect feedback trajectories, then like put those into a continuous loop, might came much, much, much later, and it was very counterintuitive to the sci- like the traditional AI mindset.

I do remember during the investment phase for, um, OpenRouter, I just didn't tr-try and re-educate a bunch of other VCs on why it was not just a marketplace. I was like, "You know what? I'm just gonna invest." Yeah. And I'm going to like take the opportunity to partner with Alex, and if not-- no other VCs get it, that's totally fine.

'Cause at the time, the-- it was not obvious, I think, to several of the investors that like OpenRouter was not more than just a wrapper around APIs. And that infuriated me, and I was like, "You know what? I don't have time to debate you. I'm just-- We're gonna, we're gonna invest." And then I think like a month later, Matt Murphy marked it up by 10X.

Like our-- I think-- I, I forget what the exact post money was and so on, but, you know, to his credit, Menlo Ventures realized, okay, there's actually much more strategic value here as well. Maybe you didn't hear all these conversations behind the scenes- [laughs]... um, but that, that frustrated me a lot.

You know, there was a lot of this like opining about wrappers, um, and if you're like, "Oh, an app is just a wrapper on a model," then like... an, you know, OpenRouter is like this wrapper on top of other APIs. This is the most stupid reductive framework. [laughs] And so it's clearly somebody who has no experience deploying products in the field.

It-it's the thing you, you dismiss other things with, like you're a wrap-- everyone's a wrapper on everything, right? Like there's, there's some point- Mm-hmm... some, some wrappers have value. I mean, investors are wrappers on LPs, right? [laughs] Like venture capitalists. So I mean, yeah, it's all wrappers down, all down to bare metal, I guess, and energy.

Yeah, there's-- when I started the whole AI engineer, I guess the coining, uh, in twenty twenty-three, like that was the, the number one pushback- Yeah... is that this is no value, you should actually just train models. Right. Uh, and, uh, yeah, I mean, obviously this is like you, you guys are one of the testaments to the fact that you can actually build, build very valuable wrappers, but also very valuable model companies.

It's so, um, hard to be like the, the day a model launches, the fact that you have an OpenRouter, um, endpoint for that model frequently at the top of Hacker News on day one, people don't realize the amount of work that goes into accomplishing that. And OpenRouter used-- like that would happen over and over again, and I remember going, "People have no idea how hard that is."

[laughs] You know, that's not- Yeah, we've covered some of the inference engineering that goes behind, uh- Yes... some of the-- with Base10 and all those. Well, today you have, you know, all those like cool code name things where people guess what OxyAlpha is and all those things. But I, I, I guess one of the things that you're teasing is how do you get that initial flywheel going, right?

Because today you have your scale and your reputation, all these things, so obviously you get-- you drive immense distribution. But when you're early on, when it's mostly- The bootstrap. Yeah. Well, how, what was the bootstrap like? Yeah. I mean, to bring it back to early Discord days, I think we like initially connected with... This is an OpenSea story, technically.

But we initially connected when you were at Discord, and we talked about like Ax- the Axie Infinity server. Oh, yes. Yes. This server was like the biggest server at the time- Yeah... at Discord. That's right. And you were kind of like constantly bumping up the limit- The limits on the server... of how many people could be in a server. Oh my God. For those who don't know, like, like ten percent of the Philippines was Axie.

[laughs] Was on, was on that server. That's the hit. It was like a meaningful contributor to the GDP of the country. It was an NFT, like crypto game. But it was- It was like a Pokemon breeding thing. Yeah. Yeah. Similar-- Yeah, there was battling, there was breeding, and, and then there was like a marketplace and trading- It's like play to earn as well.

Yeah, play to earn, and like the graphics were really cute and fun, and you kind of like, you, you, you know, you get kind of emotional about your Axie that you make. So to like start a community like that, uh, which we had to do many times at OpenSea, with basically every early project for us to create a marketplace for it, we need to make sure that the, like the community actually wants it.

Right. And it-it's kind of like building something that people want and going and telling them about it. Like you can do that on a one-on-one basis, but there's way higher leverage to do that in a community where everyone can talk to you at the same time. So we, we spent a lot of time like building things that the community really wanted. We did the same thing for OpenRouter, and you know, like the Axie community was one of like a zillion communities we did that with.

And Ansh like saw us doing it and-- 'cause you could just see people sharing OpenSea links constantly in that Discord. Like users sharing links is a really cle-clear indicator that like something important is going on. So we spent, you know, a lot of time like w- first figuring out what the gap is in the technology that people care about. Like what was the actual problem that needs to be solved?

You know, in early LLM days, it was, you know, OpenAI refusing to finish the, the prompt or like- Yes... to like complete the task. It was also- You know, inability to customize models. Um, and so there are communities that, like, are just completely blocked on that issue, and those are the communities that are most useful to sort of learn about and, and dive into and explore.

Something that really struck me at that time, you-- as, as I was just hearing you talk, I remember noting how... You, you may not remember this, but we were-- we had these, like, working, uh, Zoom calls that we were doing a sprint around for, like, this OpenSea integration with Discord. Um, and, you know, we'd, we'd get-- it would-- it was myself, my engineering team. I think you were there. And I remember, you know, um, Alex, in the middle of one of those calls, just, like, there was, like, silence.

Uh, you know, we were-- we're all like: "Oh, yeah, this totally makes sense. Let's do this." And then there's some... Every-like, everybody aligned, and Alex was like: "No, this makes no sense to me." And everyone's like... I, I remember going: "What, what? Like, the-- we-- it works. Like, you click on a link, and this-- then it bounces you out to, like, OpenSea." And he was like: "It's not a good user experience.

Yeah, we should not do this." And I remember going, you know, he was the only one-- person out of all of us to actually raise his hand and go... Yes, it made sense from a technical implementation perspective. Like, we were bouncing the user out into the-- into OpenSea, and so it kinda checked the box of the product manager's requirements on both sides. But Alex went one step further and was like: "You know what would be better, guys? If we just embedded the experience right here inside of Discord, so the link opened up as an embedded iframe, and you can just check out right there."

And not one person on the call, and like seven of us who had w-met, like, you know, week after week- And it's, and it's the guy who doesn't work for Discord. And it's the guy who doesn't work for Discord. [laughing] Like, technically, you benefit if they bounce. Ex-exactly. Right. Yeah. And that was, like, adversarial.

To keep the user inside of Discord would be adversarial to OpenSea, and yet Alex put that user f-experience first, and I was like, that's special. Wow. 'Cause it's very hard to have somebody who's technical like Alex and understands the developer flow, but also understands the best user experience and wants to prioritize that.

And that's two sides of the flywheel that, if you can get spinning, like, is often hard to stop. And you just reminded me, like, that one was one of those moments where I go-- I went-- I realized I gotta be better at user experience, 'cause I should have been the one who came up with that, and I didn't.

And I learned from you. And, um, I, I think that went into one of our case studies for the PM training program at Discord- [laughing]...which is still there. I don't know if it's there anymore, 'cause at the, uh, uh- You need an Alex is the, the conclusion. [laughing] Yeah, yeah, you need an Alex. And, and, and this is why I'm not-- you know, should-- nobody should be surprised why Stripe decided, like, they had to buy OpenRouter, because it's a really rare combination of people who understand the machine learning community, the developer experience, and the end user experience, and putting all that together has resulted in this extraordinary scale that very few other marketplaces have been able to achieve- Yeah...over the last, you know, five years.

Yeah. Well, we should talk about the other reasons for acquisitions, uh, which you- Yes, we should...which you've written about. Uh, I wanna sort of proceed, uh, somewhat chronologically as well, so that-- there, there is a point that, you know, one of the questions that, uh, Dave from HFZero sent in was: When did you know it start-really started to work?

And you brought up Mixed Draw. Uh, I don't know if you wanna bring up that story. Oh, yeah. Yeah. Which obviously- Well-...you overlap with, so- The MOA is- Yeah, the MOE was... I don't know when-- I mean, there w- there's no, like, one moment where I was like, "Oh, this is, you know, officially starting to work."

It was like moments of kind of increasing- You did do a browser extension, like, really super early on. Oh, yeah. But well, bef-- Yeah. Yeah. So before OpenRouter, I wanted to, like, explore a bring your own model experiment. And, um- Which anyone familiar with crypto is like, yeah, Phantom and all these things. Yeah, yeah.

So it felt like doing a MetaMask analogy for AI would be- Oh...kind of a fun way of exploring that. And at the time, there were no AI apps. There were probably as many AI apps that were, like, hitting AI via-- like, hitting an LLM via an API call, as there were, like, games just doing it in JavaScript.

You know, basically, like, there, there was a, there was a moment in time where it could have been the case that web apps call LLMs through the browser, like through some kind of desktop- Yes...managed app that is controlled by the user. Um, and of course, there are, like, I think many reasons that that did not happen, but back when the, when the days were that primordial, I'd built a, a, a Chrome extension called Window AI, and- With Plasmo, which I- With Plasmo.

I had come across early on, and I was like: "Who's gonna actually use this?" You did. [laughing] Plasmo had a couple, like... I think Phantom was using it. Um, there were some other, like, like, comp-- like, real companies using it. There was, like, a shim, basically. So re-Reacts for Chrome extension. Plasmo. Oh, sure.

It compiles to all these- Yeah, I see Kinda like Next.js for Chrome extensions. Next.js. Next.js. Okay. And yeah, built Window AI on top of it. The creator of Plasmo, like, con-started contributing code to Window AI i- uh, in GitHub, and that turned out to be Lewis Vichi- Oh, you're-...who is the co-founder- Of course. Okay...of OpenRouter.

That's r- Yeah, you have told me this is how you met Lewis. Yes, okay. Yeah. So, um, that, that allowed users to kinda, like, configure which model they wanted to use for a web page in their browser, and then, like, the, the app would just call out to that model when it needed to do things. You know, not the right form factor for LLMs, but, you know, it's like fun experiment.

You learn a lot, and y- like, I o- you know, open sourced it, and, uh, and, you know, the main learning is like, okay, this has to be an API, and it has to look a little bit m-- like, there has to be more of a developer experience here and more of a discovery experience as well. Like, I don't know where to use these models, and a little Chrome extension is not gonna help me discover. It's not enough real estate. I need more space. I need visuals. I need graphs. I need, you know, examples.

I need images. I need to, like-- I need to be able to, like, explore both as a human and as an agent. Yeah. So that's kind of how, how OpenRouter came to be. You know, um, a meta point that- I think is underappreciated, but Alex is reminding me, is that we were quite lucky that we were so-- we were, like, adjacent to the crypto community in those days because in hindsight, crypto ended up being kind of like a dress rehearsal for generative models, right?

If you, if you think about the, the Axie experience, uh, you know, Alex is totally right. There were not that many AI apps at the time, and while I was dealing-- You know, my job was to be the head of platform at Discord, which meant to be a general purpose place for communities and friends to create, uh, for developers to create apps and bots and, you know, other services that could be deployed across Discord.

And while eighty percent of the attention at the time was being spent on crypto, 'cause that's where all the NFT volume was, there was like twenty percent of my time, of my time I was spending with a friend, um, w- who would get Hot Bot with me and ask me for-- We'd play Magic: The Gathering on weekends, um, and he was working on a little Discord bot that could take a text input and turn it into an image, and it was called Midjourney.

You know- Was that David? That was David Holz- [laughs]... you know, he's a good friend. And David and I are both then sort of failed AR/VR founders, you know, in the last-- before that. And, um, I remember this, you know, M-Midjourney was one of the fastest-growing communities we had after Axie Infinity started to peter off. And many of the, the, like, the abstractions and the infrastructure decisions we made to scale Axie happened just in time because, you know, Axie did this and then fell off a cliff.

And then as Midjourney was taking off, we, like, explicitly decided to help David make the server, the Midjourney server, as the primary place for interaction with the, with the model because it was very hard for people to, to understand how to use the model if they couldn't see other people using it and copy them.

And so the single-player Midjourney web app on its own, like midjourney.com, had, like, terrible retention, 'cause people would show up, they'd see this empty field. It's kinda like DALL·E 2, um, and they would type in, like, cat or dog, and it was, like, paralyzing for them to have this blank canvas that they had to fill 'cause they'd never used an AI model before.

But instead, in a Discord server, you could see other people using it and riff off of their prompt, and the engagement was off the charts. And so scaling, you know, Midjourney from zero to, like, ten million monthly actives was a much smoother approach post-Axie Infinity. And so the- Don't forget the best of four pictures and you choose one- The best-- And then the other- Which is the feedback loop... the RL, RLHF feedback loop- Mm-hmm...

um, which by the way, separately, like Tom Brown, David and I used to play Magic: The Gathering on weekends, and so, like, it was one group of friends who would hang out, and we'd-- like, these concepts were all being discussed all the time. But, you know, there was... I, I, I think there were few of us who bridged both the crypto worlds and the AI worlds, and compared to crypto, where it was al-- the question was always what's the use case, you know, for this technology, there was never any need to ask that for AI 'cause it's, uh, the use case was so visceral.

It was like, I can create now anything at my-- I can imagine. I can write novels, I can code. And the infrastructure that those of us who believed in the distributed systems, like value of crypto, like the censorship resistance part, found this use case that was explosive. And I think between Midjourney, you know, the-- Claude was a, a Discord bot pre-launch, you know, that we were using internally as an LLM. Um, Eleven Labs had a TTS model that we had on Discord as well.

Like, Discord became this petri dish for, like, early apps to innovate, and I don't think it's a coincidence that they found a home there before Open Router gave the world, like, a public home store or, like, a f- um, you know, storefront. Discord was this, like, almost kinda petri dish storefront that was-- kind of had-- kind of, like, piggybacked on the infra we'd built for, for crypto communities.

And then I think Alex was one of the first people to realize, wait a minute, like, these apps need a, their own home, um, on the internet, and then Open Router, to me, was a continuation of that, of that community's needs. And of course, there was the, the crazy distribution that you enabled for a lot of these developers. So, so then my question is, how come you were-- My perception is Open Router is not that Discord-centric, right?

Uh, you have a Discord- Yeah, yeah... and you use it to engage your community, but it's not like Midjourney where, like... No, that is, like, [chuckles] the primary way people experience Open Router. Yeah, Midjourney, like, it really helps to see visually really quickly how people are using the model and how to prompt it. Yeah. And I think that is partly why the server was so critical.

It's like it is the user experience. It actually adds a ton. Yes. Um, and you can go the whole mile with just, like, prompting via Midjourney, like the, via the, the Midjourney Discord server, getting your images, and then sharing them and having fun. For Open Router, for LLMs, like, you need a lot of user experience around LLMs to make them, like, really usable. Charts. Chart point.

[laughs] And yeah, the, the, like, seeing the examples of other people is also not as useful 'cause it's a lot of stuff to read. It takes a long, long time. [chuckles] Yeah. Um, you need, like, code-based integration, not, not possible to do in a Discord server. You need, um-- Or techni- it's possible. I shouldn't say that. It's just not a great u- great developer experience.

Um, you need, like, com-- you need governance for... At, at the point where you've got code-based integration, now you need governance for managing the LLMs that have access to it, the data policies, which teams. All that stuff needs a lot more than, than a Discord server can provide. So it's just like- Yeah... it's not the right...

Well, in addition, uh, you're not wrong, but a-also there's the very important distinction that, you know, Midjourney was an end-user application. Right. And, you know, that's why Discord, which has two hundred and fifty mon- million monthly end consumers, you know, made-- it made sense for Discord to, to, to kind f- uh, to, to be a ho-host for that application experience.

What I, I knew was gonna happen soon after Midjourney found explosive product market fit, 'cause we-- I, I think when Midjourney launched, from launch to hundred million revenue run rate, it was less than eight months. And shortly thereafter, Stable Diffusion launched. And, you know, the-- all of us used to hang out in, in the Discord server. The thing is, it, it was the, um- The Stability Discord? Uh, it was the Lion- Yeah, Lion...

yeah, the Lion Discord server. Yeah, the, the image community that spawned Stable Diffusion. The image community. Yeah. Yeah. And so when Stable Diffusion came out, I realized Oh, now other people can build their own Midjourney because until then, Midj-Midjourney did not have an API, so they were a full-stack company, right?

They were training their own models, and they were deploying them as an application. But if you want to build your own Midjourney, there was no API of that quality. Um, and I think DALL-E 2 was still quite primitive. Like, Midjourney actually had great quality. And then when Stable Diffusion came out, suddenly there was this new person who could-- there was new-- this new capability in the world which is a developer could create their own Midjourney.

And that, I think, created the op-the need for something like OpenRouter because then you need an API to-- If you were-- if you had the kinda creativity of David Holz and you had Stable Diffusion as the model and you wanted to put these things together, how could you do that without having to figure out how to host the weights?

And what OpenRouter, uh, the shape of OpenRouter enabled is, is that, right? When you have open models alternatives to closed sort of applications, OpenRouter's value in the world becomes extraordinary 'cause now any developer can just show up and use the API- You just want model diversity. Ansh, did you just say the shape of OpenRouter? Oh, no. [laughing] Are you in Claude Lab? Is this the, is this the real Ansh? I've been-- I've been-- I, I'm, I'm misaligned now. I've been over-trained. Uh.

[laughing] I've been using Claude way too much, haven't I? [laughing] Claude-ish is what people would say. Claude-ish. Oh, God, I gotta un-train myself. Okay. The-- and I just wanna cap off the Mistral side. Uh, my, my, my TLDR is there was a Mistral price war is what they called it, right? Like, uh, roundabout NeurIPS was twenty twenty-three or four. Yes. December- They, they launched, uh, the Mistral 8 by, 8x7B.

Uh, it-- and, like, the price went down, like, eighty percent. Yes. To me, that's very positive because it's, like, the first, like, real competition to, to host Mistral. Is there more? Yeah, that was... I'm, like, trying to remember it, all the things that happened. It-- like, we saw that model come out and immediately saw people say that it was the best model in the world. Yes.

Like, this was, to my knowledge, the first time an open weights model was called that in real seriousness. It's hype, right? Is it, you know? [laughing] It was, it was hype. It was hype. It was also, like, hype from AI influencers at the time, and there were many examples where it was, like, outperforming GPT-4.

So people really wanted to try it out and see, is this gonna be true for me too, and if so, at what price? And, uh, the, the, like, inference landscape was really messy. Yes. We cleaned it up. Um, the-- it allowed, like, providers to compete on price- Yeah...so we could give users the best price in one spot.

And so it was, I think, the first clear example of, like, a provider marketplace working in a way that adds value to end developers. Shaun, you may not remember this, but I, I think we met for the first time a few days after Mistral came out at NeurIPS- Yeah...at a luncheon. Yeah, that's where I almost met at BFL as well. Yeah.

And Guillaume was there. I was, I was at NeurIPS at that time. You were there too. And, um, Gui-we had just announced the Mistral investment, and I remember Guillaume was over there, and I remember turning to Guillaume and asking him, like, "Is it-- is all the-- like, how are you feeling after the launch of Mistral-sevenB?" And, you know, him in his typical French fashion was like, "I mean, it's a, it's an okay model. It's not that good."

[laughing] And I was like... It was so, you know, in contrast. But I, I, is it-- I, I remember him also saying that part of the reason he felt a lot of people thought that it was better than GPT-4 was because of the speed. You know, the e-- it was an MoE model that they had, like, absolutely kinda figured out how to make super efficient. It was on the Pareto frontier.

And, and this is an important thing about LLMs, right? Sometimes when they're faster, you think they're smarter. Um, even though, like, if you did, uh, N of, you know, these co-co-co-common, like, evals that are seven-- you do seven tries, and I, I don't actually remember. I think we should go back and figure out what the data says.

But I wouldn't be surprised if it turns out, oh, on an N of seven attempts, GPT-4 was smarter on evals, but the perception of on, on like-- or correctness would be smarter or more accurate. But, you know, people, like, from a human preference perspective, felt that it was faster 'cause it was, uh, or smarter 'cause it's so fast. Um- Yeah. And actually, most queries do not take that level of intelligence.

Don't take that. That's true. Like, so this is the start of humans as router- Yes...which then eventually becomes OpenRouter as router of, of like the auto- Oh, that's an interesting way to think about it...auto mode. You know what I mean? Like- Yeah...like, because humans are the routing mechanism. Like, I will ask the fast model first, and then if, like, oh, not good enough, I'm gonna upgrade manually. Yes, yes, yes.

But then he's gonna auto it. I didn't, I hadn't thought of it that way, but that, that, I mean, that makes sense. Which then there's- Yeah...there's a lot more techniques like fusion. Fusion is a thing that we should talk about. Before I move on to those things, I just wanna close off the sort of, uh, early s- early years. Uh, one thing that I observe, which you are also an investor in Arena. Right.

And we talked about Midjourney having, like, that, that feedback loop of, uh, ABCD and choosing that very-- being, being very important. And you understand the flywheel. So how come you didn't build Arena, and how come Arena didn't build OpenRouter? Well, Arena started before OpenRouter, right? They had El- they had the school project- Yeah, LM SIS...and then it became a, a company.

Or LM Arena. Yeah. Um- So but, uh, y- and I know you had some Arena experiences, like the, the heads-up comparison type things. Yeah. But you never really went as hard as Arena did. And, and- In doing heads-up experiences? Yes. And, and El-LM SIS actually did have a router project based on LM Arena ELOs, uh, which they never commercialized.

It's hard to do a company that does both because one company is taking data and selling it, and the other company really can't [laughs] by default. So, you know, I, I think there, there is like a branding reason- Yeah...that there are two companies here. Um, like, when you set up OpenRouter, there's no training.

There are no prompts, right, aside from what your provider policy is set. Like OpenRou- like OpenRouter can't see your prompts or completions. I-if you want to see that as an org, you have to opt into it and enable it. And so we-we're like pretty conservative and careful about data policy and security. And privacy.

And Ella Marina is like, their business model is, is like oriented around the labs and, and- Because they give it for free, right? Yeah. You don't give it for free to give it for free. Yeah. [laughs] I mean, we do give some-- we, we like have free endpoints too, but like those free endpoints- We- I think we're not collecting any prompts.

We're not like monetizing the data unless you, you know, opt into it for some reason. The, this, this comparison-- I mean, you're not the first person to ask me this, and A-Alex knows this, but I, you know, I was the f-the interim-- like the found-- like first CEO- First CEO... CEO of Arena for the first five months when in-- we were helping Anastasios and Weylin kinda spin out of Berkeley.

And, uh, I did invest in that before, um, OpenRouter, but it was, it was very strange to me the comparisons that outside, you know, folks would make between the two projects because the missions were completely different. The founding entity for Arena, we called it the AI Reliability Institute because it was, it was actually there as an eval service.

Like the data, so to speak, that they, they were originally, um, kinda offering the labs was how do you m-make the evaluation of models more reliable than kinda like the state-of-the-art at the time, which was like really just finger in the wind. [laughs] Um, yeah, that's kinda what Anastasios and, and Weylin's PhD work was, as, as scientists at Berkeley, was on statistical methodologies for sort of, uh, correcting, you know, eval estimates, um, based on like intrinsic biases and how you collected the data.

Yes. Um, and so- Style control eventually. Style control and stuff like that. And which is very much like a, "Hey, how..." If, if you're a scientist and you're trying to kind of, um... The highest expectation customer for Arena was always like a, a post-training and, uh, like a, a researcher at a lab, whereas the highest expectation customer from, f-from my perspective that, that Alex w-like really understood and, and was-- the mission was to serve was, was like a developer, right?

Who then takes the result of the research and then produces an application that's deployed to the world. It's actually a completely different problem and person that these two teams were focused on. And so from the outside in-- Actually, I don't know if you remember this, but I have a distinct memory of a few weeks before we did the term sheet, uh, together for OpenRouter, I'd given you a call because we were trying to get a pooled data set together from OpenRouter and from Arena to, uh, create like an open source repository of prompts.

Mm. I mean, this project was so kinda different in their goals that it was totally- Mm... normal to me to be like, "Oh yeah, let's call Alex and see if he'd wanna team up on, on pooling data," 'cause they're so different. We need-- We, we actually don't have that kind of data at all. We-- Like we didn't have API prompts. We didn't, we didn't have like what developers want to do with the models, which is very different from what researchers inside a model lab want to do before releasing the model.

Yeah. Does, does that make sense? And so to this day, I, I think you see the, this difference, even though at a thirty-thousand-foot level you could, I guess you could kind of conclude that Arena and OpenRouter are adjacent, but, uh, the roadmaps, the missions, and so on at the time at least were like in very different sort of directions.

That ideal customer I get. I, I totally get that. Yes. Yes, yes. As a founder, I wanna own everything, right? So [laughs] like this is- That's possible... clearly the adjacency there, and I'm like gonna explore that. Uh- Own everything meaning like you don't know what to do yet, so you wanna like make sure you catch PM fit- No, I think what he says is you-... as quickly as possible...

you, you, you wanna own the entire infrastructure space, and so you'd kinda expand to whatever demands you can capture. You wanna have your play in everything. Yeah, I, I think that's, that's hard, uh, you know, in reality- Yeah... 'cause serving multiple customers is, is very difficult. Clearly, clearly, you know, this is when it looks like it's a focus, right? Yeah. Yeah.

Yeah. I, I still think f-even in the age of AI, like focus is, is- Is critical... underrated and critical. Not just because you end up with a better product by focusing your humans on it, but also because the world knows what your focus is. One thousand percent. The world can map like, "Oh, I have this issue.

Which brand out there is going to help me with that issue? This is the brand that's known for that focus." Yes. So like if I want real attention on this issue, like this really matters to me, I should go with the brand that cares the most about it. To underscore Alex's point about how important focus is, in the early days of Anthropic, uh, it, it was not easy to...

Like people think that the early days of Anthropic were like super easy 'cause they were on the GPT-2 guys who left, but it was actually very competitive. The company was starting ten billion dollars behind OpenAI. Mm-hmm. Right? And so to get to the frontier, like the big question was what, what do we wanna be known for? What's the mission? And the mission was AGI via pair programming.

And so to the, um, exclusion of all kinds of other things that were really shiny at the time, like image models and video models that were getting lots of, you know, um, momentum, the Anthropic team was like, "We just gotta focus on coding." Like that is the core capability that we're focused. And today you can see the results, right? It's a trillion-dollar company within five years.

And that focus, I think, like the high-- the focus on who your highest expectation customer is and how you exceed their expectation, 'cause exceeding anyone's expectations is hard, and doing it for multiple like customers is so e-even more difficult, um, is part of the reason why OpenRouter succeeded and Anthropic as well.

Was the focus on coding that early though, or did it come later? Um, literally from day one it was AI pair programming is-- responsibly commercialize an AI pair programmer was the seed memo. That was when I invested, right? We, we actually kinda like refined that memo a lot. Interesting memo. Well, you gotta ask Dario and Tom for permission on that.

[laughs] Um, but it's an extraordinary piece of writing that they had put together, and AI, you know, commercializing and-- responsibly commercializing AI pair program was the mission, you know, from day one. And I would say there were maybe like a couple moments in the company's history where like they did experiments to kind of see if like little detours made sense, like a general chatbot, like Claude AI, when ChatGPT was really taking off.

But, you know, at the end of the day, with especially once, um, they got their like significant pre-training compute online, I think like the, all the main evals of the company, for example, have always been Coding evals, long-horizon agentic programming. I mean, from day one that was always the plan 'Cause when, like, Claude Instant came out and Claude 2 came out- Yes... I remember the marketing mostly being focused on pros. Like, this model- Yeah...

will write better- Yeah... and- Long context... long context, um- This directly affected me- Yeah... 'cause I built something on it. Yeah. [laughs] What did you make? Uh, Small Developer, which was my Devin before Devin Oh, yeah. Yes. Yes. Small. Yes. [laughs] Uh, and, uh, you know, so I think, like, there's, there's all that really, like, good, like, focus is, is another thing.

That, that is a question that people do wanna ask. Uh, you know, you could have built any, any other things, like-- And obviously OpenRouter was working, working, working. Uh, were there other ideas that you wanted to pursue, that you turned down? You know, just the paths, roads not taken. We made a couple prototypes for things that we didn't launch.

One was a fine-tuning model-as-a-service- Mm-hmm, yeah... thing where- Lots of that. OpenPipe and, uh, all those things But it was-- it, it kinda was at a very consumery form factor, where you would give us a YouTube video or two or three. We would then extract all the transcripts from it- Okay... and try to fine-tune a model to talk like the person in the YouTube video- Yeah... or the people in the, in the videos that you sent.

So, like, a really, really easy way of creating a fine-tuned model based on, like, some kind of videos that you like. Um- That would be so useful. [laughs] We- Oh, that's-- No... we, we made it too. It was, like, kind of a- You don't think so? I-- It was, it was- But nobody used it? Is it- It was-- We didn't actually, like, test it with that many people because the model marketplace was our main, our main focus, and it, and it was, like, growing and we were building more conviction in it over time.

Um, just, just- We- Yeah. Why, why- As, as a creator- Yes, yes. I'm a creator... I've been pitched many, like, uh... You-- I have 500 hours of recorded voice of myself. Right. Make a thing of you. Uh, charge access to it. Uh, it works for OnlyFans, doesn't work for [laughs] not for us- Ah, I see... as regular people. I think all-- this is mostly, uh, u- it's just a glorified rag bot. Right.

Um, whether it's in the weights or it's outside the weights doesn't really matter, you're just doing rag on the videos, and people ultimately always just wanna find the source video, uh, that- Oh... directly answers it. Uh, my use case was mostly to practice with, from-- with myself, 'cause I often like to see what-- Like, the way I practice for a job interview or, like, if I'm hiring a candidate or public speaking or whatever is I, I wish there was, like, a good mini me- Yeah, yeah...

that I could, like, critique 'cause it's kinda hard to pull yourself out. I would never give-- I would never offer it to other people as a service. But wouldn't you wanna, like, pick your top five mentors that, then talk to them instead of talk to yourself? That'd be cool, too. Yeah. Yeah. That was- That's the character AI.

That's, that's the use case It's a replica. And that was the use case we were aiming at. I see. Is, like, you wanna create an experience- Like AI Steve Jobs and... And AI, AI Steve Jobs was the, the initial use case. [laughs] That's a, that- Even though it's not allowed. [laughs] That's a, that's a common prototype, yeah You know- Talk about adjacencies.

Fine-tuning as a service, uh, as part of the router service is something that I would typically think about as well, right? Like, like, why don't you do that? 'Cause if people are running already their inference through you, store everything, log everything, uh, fine-tune to a smaller model that is cheaper, faster, all these things that, that's within your control, right?

Uh, you didn't do that, but, like, other people would have pitched that in the general state of, uh, of a- Yes... infra startup. Yeah. I think you were just maybe a little bit early, 'cause today that's an extraordinarily fast-growing- [laughs]... segment. Like, you know, from Mistral, where they do a lot of enterprise deployments- Right...

I mean, it's tough, tough, and fine-tuning is a-- you know, custom models for ASML or whatever. And often the- But not as a router. They're, they're just like, "I come to you- Not as a router yet... because I like your Mistral models. I want custom Mistral model," right? It is not, "I want, uh, to run all my OpenAI prompts, uh, get-- store all my results, and then just move off of OpenAI," right?

They're not doing that. Uh, as a, as, as, like, a way to export off of dependency on a, on a Frontier Lab, I have not seen that yet. Right. Yeah. Which- Which was your, kind of your-... kind of position to do I mean, we, we decided-- Really, we, like, leaned into our focus and figured that, like, there are-- like we, we just saw the ecosystem develop over time, all these inference providers that, that do wanna help companies do that.

Um- Mm. Be-- Like, like, it makes sense for us to partner with them and to, like, give users lots of choice and to, like, you know, figure out what makes them, um, what gives them competitive advantages. It's, it's a whole new business, basically, and there's r- there's value in being a neutral marketplace that just kinda, like, works with those companies.

Um- Could you share a little bit, um, to, to Shawn's point, like, how you prioritized-- What, what are some ways you prioritize features? 'Cause you've always done it so elegantly that I never... You know, it just happens, and you make all the right decisions that always have product market fit from the outside looking in.

But consistently, you seem to have prioritized, you know, a lot of hit features that worked, and maybe I have a sample set bias or whatever. But Shawn's question- Can you list what you think hit features worked? What, like- Oh, the leaderboards. Like- Leaderboard? Okay. Yeah. You know, like fr-from day one- I think charting, right? Like the feedback loop... charting, BYOK. But, like, he had, like, plug-ins.

Uh, you know, he had like, uh... A-and, and I think there was a whole thing I wanna get into about, like, completions versus- Yes... check, uh, check completions versus completions, and then also, uh, let's call it, like, the, the rise of the reasoning models and how you deal with those, multimodality, all those, all those things. Yeah. BYOK.

BYOK, yeah. That was a huge one. There was one I-- Like, I think it was in early 2024. Very early 2024, we thought it might be interesting to fuse the results of multiple models together. Mm. And we launched a prototype called MOM, Mixture Of Models- [laughs]... that let you, like, pick a couple models, or we'd pick them for you, and then it would fuse the results together at the end, and it would show you all the intermediate results in this, like, big Kanban board-looking product.

What does the fusion at the end, another model? Another model. The, the, the- The smartest of the- The smartest- Of the set... of, of the three, of the set. Okay. Um- So it is like a council idea? Yeah. It was a model c-- It was like a very early LLM council. This is a multi-agent swarm as, as like they would call it at one of the Frontier Labs, uh- [laughs]... in the early days, you know?

Yeah, like some of those ideas are like going the right direction, but the devil's in the details. Yeah. There's a lot of like product refinement needed to make them really work. Um, they take your focus away from- Right... you know, whatever else you have going on, and there's a lot of like community building and learning that you need to do, and the technology might be too early.

So there are h- like all kinds of reasons they might go wrong, and in our case, the technology was a little too early. In other words, the fused result was- Mm... a little bit worse- Right. Like a Frankenstein... sometimes the same as the best model that was being used to fuse because the best model was so far ahead of options two and three at the time.

You know, over time, the top three or four LLMs have gotten closer together, still neurodivergent, but like all capable of inserting like pretty interesting ideas. Like RL has basically like expanded the surface area of creativity for machine learning researchers within each lab, and so they can, you know, diversify the reasoning power of different models more effectively.

At least that's my, my theory for why- Yeah... uh, fusion is, it like works better than it used to by early 2024. And um, so the technology was a little bit too primitive. The form factor was, was not right, and, and so we would have had to go through a couple more iterations, and so we decided to just delete all the code.

[laughs] And, uh, then years later, middle of 2026, um, or early 2026, we're like, "Let's bring it back." Like, the research is looking kinda promising for fusion. The models now have like two, three, four top frontier models that are all really good and like, like I'm, I'm frequently trying to like consult multiple models to get the best results.

Like, and then I, you know-- I, I ran a little personal experiment where I was like, I'm gonna like do a, an architecture plan for a code change. I'm gonna give it to all the models, I'm gonna fuse the result, and then I'm gonna ask all the models if the fused result is better than the individual result each model came up with.

And they all said yes, that the fused result was better. And this happened a couple times, and I was like, okay, spot check, pretty good. We should like benchmark this, and that's how we built Fusion. Yeah. Um. And it came on your Fable, so you were like, "This is Fable level." Yeah, yeah. Let's start leading up to, to this year, [laughs] which we haven't got to, gotten to this year.

You know, can you mark out the main milestones in the journey? I, I think, um, it seems like your, your promise, uh, was, you know, routing. You, you decided the business model very early. Mm. You take, you take a cut and like, you know, what, what are the major milestones that, uh, inflect the growth, right? Like you're, you're growing like nine percent week on week now. Is this, this the official number?

In terms of token volume? I think that sounds about right, yeah. Yeah. So just like can you, can you mark out like the, the sort of brief history of, of OpenRouter up to, up to, you know, the acquisition? Let's, let's call it we- Sure... we're, we're just, we're just ta- talking about, uh, you know, people are, uh, have-- y- you have a sort of your, your birth moment with, um, the Mistral stuff where, where people are really competing.

You have your state of AI thing where, where- Yeah... it's very cute. You have a hundred trillion tokens, ha ha ha. Uh, 'cause now you're doing ten a week. Uh, you know? Um. Yeah. We're doing ten a day. Ten a day now? Yeah. So, so yeah, you do this in ten days. Like, what are the major points there? You know, I just wanna like-- there's a smooth curve, but like you, you, you feel the infections.

A lot of this is, is kinda oriented around model launches. Um, we had, you know, a huge focus on pros all the way up through May of twenty twenty-four, um, because coding was just not there, and no apps were able to build much on top of it. So, um, you know, a diversity in models, but not a, not a wide diversity, and not a wide diversity in use cases. Dream Tavern was one of our top apps at the time.

The creator of Dream Taver- Dream Tavern now runs product at Cognition, Devon. Um, the-- Then we in, in the middle of 2024, we saw Claude Sonnet 3.5. That came out incredible leap forward in coding, um, and we saw the dynamics of like app spilling on top of us change. Um, we saw a huge surge in volume in like users, uh, using OpenRouter.

And, and this is when I think people started to look at the like money that they were spending and, and get a little bit like, "Whoa, what's going on? I might need to like think about like more cost efficient but equivalent models." And shortly after that, I, I think it was after Sonnet 3.5, Mix, uh, Mixtral 8x7B came out, and everyone was like, "What?

This is the model." Like, the Open Weights community delivered. And so it was really good timing from Mistral. Um- Basically all of Angie's port codes are just helping you out. [laughs] It takes an ecosystem to grow an Open Router, you know? Yeah. That was the-- Yeah, it was. It wa- like it was li- this early ecosystem, it was like a, a swing action where like model labs would come up with some sort of frontier innovation, like usage would surge, then users, you know, look at their invoices thirty days later and like, "Whoa, what's going on here?"

And then Open Weight models would deliver like a, a like co- cost effective options two, three months later. We saw that happen several times. Uh- By the way, one thing- Market. Yeah... one thing you also did with the coding agents was that you broke out which are the top coding agents, and they love that. They love that leaderboard. The, the client versus the root code versus the what have you.

Yeah, yeah. Like client was like the top of our leaderboard at the time. We, we then at the end of... and I'll skip forward a little bit. The end of 2025, there were quite a few coding apps on the leaderboard, but they were all IDs or, or, you know, terminal-based- Agents. And at the end of twenty twenty-five, we saw OpenClaw appear. Mm.

And OpenClaw was, like, particularly interesting because, one, it was like a, a new form factor that, like, brought in a new type of user, not just a developer, but like a, a productivity or sort of a, like an internet creator came to AI for the first time. Mm. And it also had an interesting architecture where it was, like, calling your chosen model for these heartbeats to see if it was still alive, in addition to actually using the model for real tasks.

And the heartbeats are like they're kind of- Frequency... you don't wanna pay a lot of money- Every thirty minutes... to do a heartbeat. Yeah. So, um, the, the auto router that we provided was really, really useful to this, like, wide range of users all of a sudden. And, um, and so we just saw it rocket exponentially, and then we saw, you know, like OpenClaw just blow up and a couple other, um, apps lean into that new paradigm and do something similar.

Um, Hermes came out and really leaned into things like the auto router and built like a really good community and leaned into like, like basically skill management and making it really easy and effective for people like, like set their memory in the agent and- Yeah... and build really good skills. Which another thing you never did, memory skills, sandboxes, all these like adjacent things you could have done. Could have, but like- [laughs]... it-- I think like- It's hard to bet...

they're also very-- there are things that developer-- that really matter for like the developer use cases that were coming out at the time. Like, developers wanted to architect those things. Right. Those were kinda critical to building a good user experience. It's re-- it was like hard-- It's been hard for companies to find abstractions that work for all developers on the memory layer.

It is, it is-- You know, there are some, um, like Mastra has done a pretty good job, for example. But, uh, but like developers have like lots of varied preferences for them. And then we saw, you know, our, the way our, our leaderboard has, has changed over time is kind of like a movie of how the AI space has changed over time.

If you just sort of like go to the wayback machine and look at the, the, the rankings leaderboard and the apps leaderboard over time, it sorta shows you like what's happened in AI over the last couple of years. To me, the coming-of-age moment was, uh, Andrej Karpathy was like, "I no longer read LocalLlama 'cause like I just go to OpenClaw-- uh, Open Routers leaderboard." [laughs] Which I, which I remember that. Yes.

I think he probably like said like, "Sorry guys, [laughs] like I'm gonna send a bunch of traffic to you." [laughs] So I, I was gonna bring it into the Stripe, uh, thing. Mm. How does that kinda conversation start? We had this long-standing relationship with Stripe, though, from, you know, like many different projects that we had worked on with them. We invest, you know, a lot of effort in countering abuse, um- Token fraud...

and token fraud. Can you give some numbers just to, so people under-understand? I think I like, I, I posted about this. We, we blocked ten X as much dollar volume last month as the month before, and the types of token fraud are diversifying quite a bit. You know, there are like fraudsters going after typical stolen credit cards, but there are also, you know, people trying to resell traffic against the terms of service. There's like hacked accounts.

There's people who just lose ac-- you know, like their, their, their whole company is compromised, and they don't even realize it. Um, and we help them like regain control and detect it. There's-- [chuckles] There are accounts that are like reselling inference on the side. There's-- there are accounts that are dealing with, you know, a, like an accidental runaway agent, and, and they don't realize it.

Not, not a hack, but it's something that blows up, and, um, and the company doesn't want it. And so our trust and safety team like works a lot on all of these like categories of problems and he- and helps block it and detect it. And so we've built these, you know, we have models around them. We have-- We worked closely with Stripe for a while on this, and I think it's gonna become a huge problem in the ecosystem.

Like, we're already seeing a lot of companies start to see these fraudsters like spread and look for other ways other, you know, other than Open Router to other fraud vectors. And if you're making a gateway or, or selling like generalized inference, you are a target for fraud. If you're selling very discreet, like intelligence products, intelligence products that are like doing something pretty specific, but not like, you know, just reselling inference with some added capability- Yeah...

then you're way less likely to get these fraudsters. So it-- I think we'll see companies also move away from just reselling inference with some sort of like added capability, and move towards sort of like discreet tasks and charging for those tasks, and charging for those enhancements, and letting people bring their own inference f- like in a third-party way.

Whoa, okay. A-and yeah, obviously, you would power that. Uh, but you would- Right... people pay, uh, for outcomes or per task? I think people will pay, um, you know, I think like the, the, the Datadog pricing page is a, is a good look at like the future to come. It's like companies, like infrastructure companies will like charge for different types of events that they're providing, and there'll be lots of like continuous pricing models that look like that.

And of course, there will be like if you go down, uh, towards consumer apps, you know, simpler pricing, s- more subscriptions, um, you know, fewer events to worry about, um, and ones that like are not- Focused on just adding a markup on top of inference. Yeah. It's g- not just because fraud is hard, but, but also because the pressure from the labs and from infr-- like good inference providers to, like, do a commit and then bring your inference elsewhere is gonna be very high.

Any comments? Uh, two. One, you know, I, I think Alex has done a very eloquent job of describing something, you know, counterintuitively I knew would be a thing at scale like four years ago because of Discord, and the particular experience that taught me this was, um, you know, sh- as we started scaling Midjourney, you know, the one of the primary ways that we used to give away or like get people to try Midjourney early on to get to their first ten generations, 'cause, you know, ten generations of...

Ten images generated was y- roughly the magic moment activation point we found. Like, once you'd done ten, you were like, "This is extraordinary." Um, but for that we-- So we had a free trial with Midjourney. And one day I woke up, 'cause I was the head of platform and had to monitor, I had all these dashboards, um, and I had like three missed calls from David, and it turns out, like they had, there had been this flood of new users overnight and, and we were like, "This is great."

And he was like, "No, actually, we shut down the free trial." And I was like, "Why is that?" Um, and he said, "Oh," and he looked at the geolocation IP addresses, and basically somebody in China had started to resell Midjourney free, you know, subscriptions with the free trial as a way to like, basically, you know, it was fraud abuse, right?

And was that- Even for a specialized model like Midjourney. Yeah, and that was an act- actually an application. So this idea that... I, I think the big picture realization I had back then was, hey, there's a new type of unit of value that's being streamed across the internet called a token, and over the next ten years, the entire internet value chain was going to have to deal with the fact that, like the more valuable tokens got, the more bad actors are gonna go to try to get their hands on those tokens.

And anytime you scale something and the payload gets more and more valuable, more bad things, people try to get access to that value. And, and so it, you know, it was very obvious to me back then. And so, look, to this day, I don't think there's a free turn... Like, I don't think Midjourney's ever actually turned on the free trial since then because it was really not an easy problem to solve in terms of trust and safety.

Uh, that's why I try to start teaching the class Security at Scale at Stanford. Like, this is l- like one of the... That and the anthropic learnings, uh, to me it was clear that the need for security at scale was gonna be enormous a few years from then. 'Cause if you just do the math, right? Think about like if we're, you know...

Online payments, you know, has started roughly in the '80s and '90s, right? And grew to over a trillion dollars over the next ten years, and we needed to build entirely new payment solutions to deal with online fraud. Um, where we are today is roughly there on tokens, but over the next even five years, we're expecting the token economy to get to like roughly five trillion dollars, and over the next ten years, I'd be shocked if we went to ten trillion dollars of token flow.

And so if we were starting to see such aggressive abuse and fraud at sub-scale, Midjourney, remember Midjourney at this point was like less than three hundred million revenue run rate a year. I, I just realized we were gonna need like entirely new like systems to deal with the fraud that was gonna happen for trying to get into the token flow. And so, um, my, I, I...

You know, when I, I forget the board meeting it was when you brought up that, you know, Stripe wanted to, to partner up and it made so much sense to me 'cause Stripe Radar... When I was at Kleiner ten years ago, we invested in Stripe and the whole pitch that, you know, Patrick and John communicate so eloquently was like, "Hey, unlike traditional payment tools like Braintree that do a seven-day verification like KYC and AML to get, get the fraud out of the way, we actually just bite the fraud cost up front as customer acquisition cost and give-- tell a developer, like just use five lines of code and we start accepting your payments f- in five minutes, and what'll happen is over time we collect all this data on the developers"- Ah, Cloudflare model.

Is the Cloudflare model, right? [laughs] And, and they did. Five years later, they launched Stripe Radar and Stripe really today is a security company. That's the real... People think it's a payments company. No. The reason... There's lots of other payments providers today that give you like cheaper pa- payments transmission, but the reason Stripe keeps, you know, being the dominant one here in Adyen and Europe is 'cause they have extraordinary fraud detection that they've built w- you know, with over the years.

It's the same story with Elon and Max Levchin and, and- And upfront, yeah. [laughs] PayPal. So, you know, I think the story shows up over and over again, where every time you have value streamed across the world in large amounts, you need new protection and security infrastructure to fight to, to keep the bad guys out and allow the good people to like have their transactions happen really fast.

And so I think, you know, the t- this is why the, from my perspective, like the Stripe and OpenRouter story is a security story for the internet ecosystem, for the frontier AI ecosystem. Without a partnership like that, it becomes very hard to defend the quality of experience and the speed and all the good stuff without letting the bad guys get in the way.

Um, the second is that, you know, the, there's this underappreciated thing about like the fact that you need to... Like these, all the bad things that, that Alex described as being perpetuated by humans right now is going to be perpetuated by AI agents over the next ten years. Oof. Right? So think about the like recursive scale we're about to see of bad actors. It's not just bad human beings, it's, it's all the bad agents that are gonna be attacking the token flow. And, and there's...

It, it's very hard if you're a researcher and an, at an AI lab to reason about that problem because the only data you have is how agents you're training are going rogue, but that's just a fraction of all the bad behavior on the internet that we're gonna see. And so what you need is, is defenders, new sheriffs in town, which cowboy hats, um, that can, can see all the bad behavior from AI agents across the ecosystem from different model labs and different post-train deployments and different developers and take all of that data and say, "We're gonna build a shield for the entire token economy."

Because without that, you know, the amount of fraud we're gonna see of this ten trillion dollars in GMV and global GDP growth is, is like a huge percentage of that I think is going to be fraud, abuse, and we might never get there if people just don't trust tokens, right? Um, and I don't think this infrastructure exists, so you have your work cut out for you with, at Stripe.

But I don't think people have realized the scale at which agents, agent, agentic fraud, like bad behavior perpetuated by AI agents, is about to hit us like a tsunami. Yeah. Uh, I mean, there's a lot to dig into there. Uh, I want to give you the last word. We do have to wrap. Um, um, what can people expect from O- OpenRouter and Stripe?

I mean, I think this is a really good way for us to accelerate go-to-market and, um, to go upmarket more quickly. It's also, you know, as Ansh eloquently described, this is, there is a really clear better together story here when it comes to improving trust and safety and making it really easy to, like, accept tokens and let people bring their own inference to your app, and to help developers just, like, build on top of inference, uh, going forward.

We have a really strong brand with OpenRouter, and we're keeping the brand. So, like, OpenRouter, like, as a, as a product and the roadmap and, and the name and the brand, like, you know, is staying the same. And so what, like, you should expect, you know, in the s- next six months is that most things will be like what we would've done had we been independent, except everything will be moving faster.

And that, that's kind of like our, you know, near-term goal. Longer term, hopefully I can comment on it soon, but I can't now [laughing]. [laughing] Okay. Well, uh, we'll hopefully do a follow-up at some point. Uh, but thank you for being so generous with your time, and, uh, congrats on the partnership. I mean, this is one of the beau- most beautiful bromances I've seen in AI. [laughing] Just starting out.

Starting from Stanford- Just starting... to here. Yeah. Lots more to do. Yeah. Lots of sheriff sh- sh- policing to do of the- Yes... of the, of the token- The cowboys in town. Yeah... the token economy. New-- we need new- Mm-hmm... we need new sheriffs, for sure. Yeah. Awesome. Thank you. Thank you. [upbeat music]

OpenRouter: from Seed to Stripe — with OpenRouter’s Alex Atallah & AMP’s Anjney Midha

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin