Who Checks the AI Labs? Independent Verification with Andrew Freedman

0:00 Welcome back to the AI Power Podcast. I'm Greg Allen. And today we've got Andrew Friedman. I previously sat down with Andrew Friedman a year ago on my old podcast, the AI Policy Podcast. And Andrew was leading an organization called Fathom, which at the time was a relatively new but surprisingly fast-growing, surprisingly influential AI policy organization.

0:25 Now, more than a year later, Fathom has had a remarkable string of successes. That includes having multiple draft pieces of legislation that they contributed to turn into actual pieces of legislation passed at the state level, and some of their ideas reflected in some of the highest profile AI pieces of legislation that are getting bipartisan support in Congress today.

0:49 So I thought it would be a great time to bring Andrew back, figure out what's going on with Fathom, and get his sense of the state of the AI policy debate. So, Andrew Friedman, now the CEO of Fathom, welcome back to the show.

1:02 It is such a pleasure to be here and such a pleasure to see you again, Greg.

1:05 So can you just help us understand like what is Fathom? What problem was it created to solve? And why did you structure yourself as a nonprofit as opposed to a political action committee, a lobbying organization, an AI company, an insurer, you know, whatever.

1:21 Fathom at its heart is a policy creation organization, a policy advocacy organization, and a coalition builder. And we were created with the mission that AI was going to disrupt absolutely everything in the world, and that the only real ways of being involved in the debate to that date were either being a part of the AI companies or that you were really married into the effective altruist movement.

1:46 And our vision that we had in this is this is going to affect everyone. So creating that platform where we could create far more voices to come in that would end up being a power source, but also an idea generator for what was actually going to work for the transition to AI. The reason we didn't become like a PAC or a political action committee or that we don't do really partisan things is in order to build that big tent and that vision for the rest of the world, the earlier you fall into a camp, the more you exclude yourself from other people.

2:16 So we really tried very hard to show that we were not camped in any one place, but that we hurt everybody. It's not that we don't worry about existential risk. It's not that we don't worry about the same things that the AI safety world worries about. And it's not that we don't hear from the leaders of the business community as well that we have to remain innovative and we have to be globally competitive.

2:37 We need to be able to take in all of those voices and create solutions that work for everybody.

2:43 Yeah, and I have had the privilege now of attending your Ashby summit two years in a row, which is, as you said, kind of a big tent event. It's remarkable the diversity of viewpoints and positions and stakeholders that come there and have a very fruitful and productive conversation. So we're gonna get to all of the major policy positions that you've staked out and that Fathom is now advocating for.

3:10 But before we get to that, I actually want to focus on one of the more interesting things that you folks do, which is you actually run polling, and it's not private polling to more effectively get your agenda passed. You actually publish the results to inform policymakers, to inform the public, to inform the companies just what the American population wants from AI.

3:32 And you had a survey come out in April 2026. And I want you to just go through what were the major takeaways, what were you trying to learn, and what did you learn?

3:42 First of all, not only do we do the polling, but once a year we also sit down with focus groups from all walks of life, which I have to tell you, there's a whole bunch of new ways of doing different stuff. You know, you can have a synthetic focus group or you can have a synthetic poll. AI is now taking over a lot of insights work, but nothing changes the way I see the problem more than hearing from Ohio MAGA voters for an hour and a half about like when when you do more than just ask the basic questions, but you actually are having a dinner conversation with them.

4:10 Where do they go by the end of that? Right. Like what goes past their initial knee-jerk? And instead, when they're like, well, okay, when I really dig down into it, this is what I hope happens. Or I'm like just overly skeptical that this will ever come out this way. And so that has ended up, and we published those things too, and but it by far opens my mind more to like what we think the political conversation is in San Francisco or DC is nowhere near what the country thinks the political conversation is.

4:40 Oh, that's really interesting. Because like a poll is what you think when you're asked on the spot, and a focus group is maybe one of the sources of that belief, or if you really thought about it for a while, what beliefs might you come to realize you actually hold? So I think that's super interesting.

5:00 So just walk me through some of the big takeaways of the polling and of the focus grouping.

5:05 Yeah. So the poll, honestly, what's so interesting about when we took that poll is it's actually, it was actually before the whole world did actually start to shift on AI. But what we did in that poll is uh really set up Steel Man versus Steel Man. So instead of being like, do you care about safety or do you care about beating China?

5:23 is like, would you take a delay in our models being released if you understood that it would have a significant hit on GDP, but had a significantly better chance of not having bad outcomes with children? And what we found was uh across the board, there was a real feeling that there wasn't enough vetting going on before and enough uh of just guardrail work uh as it went out, even if you really stacked it up and like really asked the tough questions of what it would cost.

5:54 The one place that started to break down was the bipartisan nature of it broke down when it came to uh competition with China versus being a global leader. Democrats are interested still in being a global leader on how AI should be governed. Republicans, that is not a priority. And instead, it is oh no, we do need to remain competitive with China.

6:17 And if something really cost us that race, then that would be a difficult pill to swallow. However, the end of it really does come back and say, But trust has to be part of that infrastructure, right? That if if this stuff comes out and there's no trust in it, then we've lost the race with China anyway.

6:35 And so a really nuanced, complete picture of where people would make the trade-offs at that point in time. We will be we will be doing one again, uh, which we will share, we'll share with you, Greg, first, uh, probably more in the new year. And I think I'm gonna be really interested to see how the how the world has shifted between then and now.

6:54 But your polling in some way gives a guide to the present moment. Because as you said, you asked the steel man question of do you think there's enough vetting going on, even if right, there's all of these downsides and trade-offs to vetting? And that was before we had major safety incidents that we've had now, such as the hugging face attack.

7:18 So, what your polling indicates is we've got a very skeptical population. Where do you think this is gonna go if you introduce a very important new fact pattern to the debate? Well, you end up where we are, which is AI is monstrously controversial, even as it's achieving billions of weekly average users.

7:38 So AI is a part of people's lives, but the political support for AI is still very much in doubt.

7:45 Which I'm sure we'll get to in in a little bit. Well, we were pushing this idea of IVOs when no one wanted to talk about IVOs with us. And we were I talked about it with you, but yeah. Yeah, you would talk about it with me. But we were in the wilderness for for a while. And it it is what we had seen in the in the in the data.

7:59 We're like, no one's reaching for this answer yet at the political level, but the second something happens, they're gonna start reaching for it. And we really did end up in this place where we're like, okay, here are the two worlds. Nothing ever happens, in which case you're right, we probably don't need a different regulatory system.

8:14 Or we start to actually see impacts of AI. And then people are gonna really start getting hungry for solutions. And at that point, we've already set the table for what the solution needs to be.

8:25 Yeah. To use an old political term from the financial crisis, you want shovel-ready projects so that when the crisis occurs, you've got some kind of policy proposal that's ready to go, and you've you've done the work of making this policy proposal increasingly mature as we enter what a lot of people think is the crisis era of AI.

8:44 Yeah, that's correct. So I want to ask one thing that I thought was really interesting from the poll that I saw. I mean, some of it makes sense, right? 90% say child safety is very important. 90% say AI products for kids should be verified safe before use. 86% say trust and verification is important, even if it slows innovation.

9:08 But one that I thought was pretty surprising, uh, maybe not so surprising, but nevertheless revealing is maybe a better word. Who should lead the AI transition? Who do you trust? Independent experts, 71%, tech companies, 61%, federal agencies, 51%, elected officials, 37%. What do you make of that?

9:33 I don't think that's a surprise that our institutions are less trusted than they than they ever were before. That is a thing that's heading in the wrong direction. I would say the actual surprise in there was how high tech was in that poll. It was a little out of line with some other polling we've seen from Edelman and some things we've seen since then.

9:52 So, you know, always with a poll, you take one data point and you kind of say put it in the forest. But I do think that it does represent a little bit that everybody recognizes that that's where the expertise is right now, and that that there's a little bit of like, if they can't get this right, what hope does government have of getting it right?

10:09 So we'd rather see the labs trying to figure out how to get this right than we would want to see a direct top-down government world on this. I mean, that was kind of the birth of IBOs anyway, where there was like a very real political question of where we could uh gain trust on it. I would be interested and will be interested to see uh where that number has gone now that AI companies look fallible, right?

10:33 Like it's you are starting to see like I wonder if if that number has started to shift between government and um tech.

10:40 My guess is it's closer to parity than it was when people wonders if uh independent experts like yourself and myself are are still commanding 71% trust ratings in this.

10:52 Blue Rose did uh did a poll a little while ago where it essentially asked about I like IBOs or or independent experts, and it was 76%. So I do think more and more people are like, there has to be another solution here. And so where what do we reach for?

11:06 That's so interesting. Okay, so now you you said that the focus groups were some of the most insightful part of the conversation. What did you learn from the focus group specifically that that wasn't captured in the polls? A few insights that were there.

11:21 One is if you talked about tech in general, they'd be like, I do hope that tech finds a way. When you started to talk about specific tech people, or even just like all the leaders were named, it had a visceral reaction. So if you were like, Do you hope AI companies start to do this? And then like, do you hope uh Elon, Dario, Sam, Zuckerberg, then suddenly, no, I don't think anything that they will do will be in my interest.

11:46 And just watching them disseminate the difference between uh tech experts and the tech oligarchy, really like it would they would always be kind of of two minds. Like, oh, if the companies don't get it right, what chance do we get? Wait, but if I view the companies as those people, there's no chance we're gonna get it right.

12:04 Yeah.

12:04 But then is there an archetype for who these independent experts are? Is it like a Bill Gates? Is it like a Joshua Bengio? Who are these independent experts that people seem to be reaching for?

12:17 When they have that discussion, it is interesting. It's not just that they hope that the like best nerds from the labs come in, which I do think that that's part of it, is like, hey, whatever the engine is that is fueling the anthropics and the open AIs of the world, we hope some of them leave and go started.

12:34 But we also hope that they're starting to fill with experts that we care about. So that um that this is built not just with those people, but but maybe like child psychologists and people that actually understand what happened to children uh or mental health or the economists that actually understand what's gonna happen to jobs.

12:51 When we talk a little bit more about what that might look like for more of the like X-risk, it gets like a little bit more fuzzy with them. They're just like, I hope it's a real biological expert. I hope that uh that um those that know the most about loss of control are there. Um they don't have like a particular archetype in mind for who that person is.

13:09 So I'm gonna just go wildly into speculation right now and say an archetype that I think that there's a non-trivial share of the American population that might be reaching for. Um, have you ever seen the Michael Bay disaster masterpiece, Armageddon? Yes. So, you know, there's this scene where like NASA is talking about how are we gonna stop this asteroid from destroying the Earth?

13:38 And they realize we're gonna have to drill into the asteroid and nuke it. And they bring in Bruce Willis and his oil rig crew as the drilling experts. And they're talking about all this, you know, difficult space travel stuff. And Bruce Willis is like, yeah, maybe you know all about your space mumbo jumbo, but you don't know nothing about drilling, right?

13:58 And he has this sort of like folksy everyman, but but high degree of competence and expertise in his field. And I just wonder if that's it, right? Like there's there's not some number of Nobel prizes you can win where there's this tranche of the American public that is willing to trust you. Um, but there's there is this valuation of expertise, but it's expertise that that resonates with them.

14:22 I I think there's a real there there on that, because I do think people don't see themselves in the values, the values being chosen at this point uh for AI. Like I think they they continue to say like it's people that I don't feel any sort of kinship with telling me that as there's a utopia. When they tell me what that utopia is, it sounds foreign and awful.

14:46 And then they tell me that also I should be worried about like jobs in the end of humanity. And they there's yet to be a person that I really would want would want to have a beer with that has sat down and said this is what this AI system would do, and I've looked under the hood and it looks good. Yeah, and and so I don't think you're wrong about that.

15:06 I uh I I think that there is like a real lack of anybody who presents a vision that anybody likes at this moment and uh and then can say, like, and I I'm talking to the nerds, and let me tell you that it's going in the right direction.

15:20 Yeah. Yeah. All right, so I love that. Let's shift gears now to the policy proposal that I think Fathom is rightly most strongly associated with. There's obviously a lot of different aspects of AI policy where Fathom is a meaningful part of the conversation. But I think the one that you've pushed the hardest and the one that you're best known for is an IVO.

15:43 So, what is an IVO? What problem is it trying to solve, and what's like the mechanism by which it tries to solve it?

15:49 So maybe the easiest way to talk about this is that um from the beginning of any emerging uh uh technology in the world, it is most often not governed directly by government. Uh and that uh that's the when we start doing work on what is what's the right form of government for for AI, when you go and look at the beginning of consumer uh electronic goods, when you go and look at the beginning of the railway, when you go and look at the birth of uh flight, a lot of this stuff had to start in more technical neighborhoods where people were trying to do things, failing at doing things, figuring out ways to make things safer.

16:29 And uh these more independent third-party groups would spring up. And then when you also do a dive deeper, you see a whole bunch of failure modes of when that happens and a whole bunch of successes. Um uh and like to name a success, Underwriter's lab, UL figured out very quickly how to not make uh consumer electronic goods blow up people's houses.

16:51 People's houses were blowing up. If you got the UL stamp, turned out that your house didn't blow up and UL was like.

16:56 What decade are we talking about where consumer electronics are blowing up people's houses? Let's see.

17:00 This was though uh Chicago's World Fair was when uh uh was when consumer electronics. So late 19th, early 20th century. That that that was the birth of UL was the Chicago's World Fair.

17:11 And and sorry, just like like walk us through like what did Underwriters Lab do and why did it work?

17:19 Underwriters Lab was uh a group of experts who actually came up with technical specifications for understanding everything from like width of of insulation to um to how to like how to make sure that your circuitry wasn't gonna wasn't gonna have a short in it. So there it was just a long list of technical specifications that they could keep up to date with electronics in a way that that couldn't be done by government, right?

17:44 There was no list, there was no way to keep government up and running on that list as these things were being invented. And they started to be involved with insurance and they started to be written into code and they they became a vital part of the system such that UL became synonymous with safety.

18:01 And I love this story because it's it's one that I often use in the conversation of AI safety, although I hadn't really been talking about the UL connection, so I'll have to add that to my spiel. But if you think about like the history of electricity, there's this book that I love called The Age of Edison, and I struggle to remember the author's name, but it really made an impression on me.

18:21 It is about the early decades of the second industrial revolution, the electrification revolution. And what's amazing is that right now, if your friend has a bunch of candles in the house and they have a bunch of light bulbs in the house, and your friend's house burns down, everybody's first thought is it's the candles, right?

18:41 But in the late 1800s, that was like not such an obvious win for electricity. Like electricity is not safe because of the laws of physics. Electricity is safe because we did a lot of hard work to make it safe. Um, and we figured out that, like, as you said, if you're gonna have this voltage and you're gonna have these amps, then your insulation needs to be this thick, right?

19:07 And your wires need to be such and such number of inches far apart from each other, so on and so forth. People had to learn these standards, had to understand them in a, you don't build it unless it follows these rules kind of a way. And that is why electricity is safe. And so I often think about that in the context of AI.

19:28 People ask, is AI safe? And my response is that is really a question about us. Like, are we going to make it safe rather than it is about the nature of the technology on an inherent basis.

19:42 I couldn't agree more. That it is it is a race to science and a race to the incentives to make sure that it gets put everywhere it needs to get put.

19:51 Uh yeah, it's like it's like um technical know-how, process know-how, and process dissemination followed by process discipline. You know, like those are the elements, and you have to chain all of them together to get where we've gotten on electricity, to get where we've got on all these other industries that are now pretty safe.

20:12 And I think I think the challenge is the general nature of AI, where before you could maybe say, here is all of the really foreseeable harms from this thing. And and you can with AI, it's just a very, very, very, very long list. And um, and even then, even if you thought you listed all of them, which they all do, we might be surprised with what the harm becomes.

20:34 And uh and that in our mind justified really taking what has been uh a structure that has has uh shown up kind of haphazardly during an emerging process, like where it shows up, it shows up, and really starting to say, like, okay, if these third parties are gonna end up being the central way we regulate these in these critical opening years, how do we make sure that they're actually experts in what they do?

20:59 How do we make sure they're actually independent from those that they say they're checking? And then how do we hold them accountable and keep the ones that are doing a good job and boot out the ones who are doing a bad job? And in essence, that's the system of IVOs that we've created here.

21:12 Um so, like if you think about an underwriter's labs, they're they're doing research and development, labs, right? They're trying to figure out what makes stuff safe. They're running tests to figure out what's safe. And then they're also part of creating this incentive structure whereby the people who follow their recommendations are glad they're doing it somehow, some way, they're making more money because they do it.

21:36 So now connect that to IVOs. What is the IVO? What is it doing? And then how does it drive behavior of the organizations in the ecosystem of AI such that they actually want to do the stuff that would make the world safer?

21:53 Yeah. So IVOs come to the government with a scope of harms that they think they are particularly good at holding account. Accountable and tracking and understanding how to keep into an acceptable level of risk. And they apply essentially to which will be the state of California, for instance, and say, here's how we know we're independent.

22:12 Here's how we know we are the right experts and we've done the right work. But most importantly, here are outcomes that when that we can know we can actually track and monitor and hopefully mitigate some of these issues. And you, government, should designate us IBOs. At this time, that counts as evidence in law that you are doing some form of reasonable care.

22:31 It doesn't give you any formal liability shield, but it does form the basis of you showing that you really tried to do what was uh in the the best, uh the most reasonable care for the for the okay.

22:43 So so let's just back up and take that step by step. Yeah. IVOs are independent verification organizations. They are verifying some aspect of an AI product or an AI process or whatever as some degree of safe or at least abiding by safety practices, processes, technologies, et cetera. So the IVO goes to the government and says, I'm really good at assessing models for safety in the context of interacting with children, right?

23:18 Like I'm really good at testing models to see if they have all the good safeguards that you need in place, if this model is ever going to interact with kids under the age of 16 or something like that. And the government says, we have the way of verifying that the IVO is doing a good job. And we bless this one and we bless this one and we bless this one, but this one needs to come back to us with a stronger, stronger pitch.

23:42 And then the IVOs, they do what? They go pitch themselves to the AI companies and say, give me money so I can certify you, or how did they interact with the private sector side of the equation?

23:54 So currently, the first step in this legislation has been just the designation is the designation, and that serves as a strong signal. And those who feel like they need that strong signal will use it. Our theory has always been let's figure out what the group of IBOs will end up being before we start saying, like, and here's how they are necessarily used or or or not used.

24:15 I will say something that has shifted over the last year. When we first came with this, it was like this can be an entirely voluntary system, and you allow the risk to show where this needs to be. There'll be liability risk and reason to pick these guys up, there'll be signaling risk and a reason to use IVOs, but maybe less of a direct government mandate.

24:35 Over the last year, especially for catastrophic harm, we have just seen more and more of a push to say from the labs themselves that maybe this should be mandatory. And uh and that that window I think is pretty permanently changing and a place where Fathom is is really seeing that, which is that Yeah.

24:55 So I I think your your hypothesis that that the IVOs would generate this um re self-reinforcing loop of of virtuous cycle benefits was not an unreasonable hypothesis. That is empirically what has happened in many industries. I think you're just facing the astonishing exponential growth of capabilities of AI and the timeline that puts you against.

25:23 I mean, I think the voluntary system would have worked if we had a decade, a decade and a half for all of this to shake out. Um, given that right now we're we're facing some non-trivial risks of some catastrophes in the next six to 24 months. Um mandatory is the path to making all of this happen faster.

25:46 Is that true?

25:47 At least for parts of the ecosystem. And we want to be clear, the whole ecosystem matters. You know, everything from the delivery of an AI product to a consumer that is reliable in the ways it needs to be reliable. All of the ecosystem needs to grow. Um, uh, and it needs to be a fairly robust ecosystem in order for enough capital to flow in and and the RD to be to be done.

26:08 But for really a lot of the frontier model work, it's just becoming clearer and clearer, especially when you talk about loss of control, crypto, uh um cyber capabilities, CBRN risks that that we are talking about uh kind of jumpstarting that system a little bit more with uh with some mandatory controls.

26:27 Got it. And and so how does like an an IVO ecosystem um exist alongside or exist in place of organizations such as the US's AI Safety Institute, which is the Center for AI Standards and Innovation at NIST, or the UK AI Security Institute? Like, how does the the IVO universe coexist or not coexist uh with the government trust ecosystem?

26:57 I definitely think it is uh belt and suspenders, although let me modify that a little bit more because I'm not just saying it's a redundancy. I do think that uh that it makes sense for government to become experts in this world, um, uh, not only for kind of directly going into the labs and understanding where the labs are, but also understanding the IVO ecosystem deeply.

27:20 That there's a lot of reasons that we need to grow uh the the safety institutes, uh Casey, um, NIST uh and and ongoing um evaluation work within the governments themselves. But the the tech debt that's here right now is far beyond what we could grow in a fast manner inside the the government itself. Uh I think that there's there's some magical thinking happening still where people are like, well, if we just tell the government to do it, they will be able to do it overnight.

27:54 And it's like the answer is no. Like the for a number of reasons, some of them are entirely bureaucratically boring, but it's just true. Like you're not top talent, might like some top talent might decide they've made enough in the world and they're just gonna go over and and be a part of the system.

28:09 That is not, they're they're never gonna grow an army of safety tech experts from those that have just decided they've made their billion somewhere else and now they want to, they, they wanna um join government. Government is not gonna find a way to be able to suddenly be the world's best IT department on this on this stuff.

28:27 But there are places where I think that only they can fill, right? That they have it's it there's no amount of classified work they can give to a third-party vendor that can can bring them in.

28:36 Um and then again, also like Yeah, there's just there's just certain certain sections of national security where it's it's really tough to imagine anybody but the government itself doing a good job.

28:46 Correct. And and I do think that also uh, you know, that the government having more full insight into what the capabilities of these systems are um before release and as they're being released, uh also allows them to say, like, oh, we have a problem in our financial sector, and that go that needs hardening right now.

29:07 That you know, that that it's not just did you put the right guardrails on guys uh on the on the system? It's like this is gonna, even if you put the right guardrails on, we now have a new problem in the world that that that we need to get the rest of the the world to respond to. And that will still be the government's job.

29:25 Like, hey, like we need to harden parts of the internet, we need to harden parts of our financial systems, we need to better understand what's happening in our you know, biological wet labs uh in order to make sure that some of these things don't come to pass. That is gonna be a problem that only government can can help solve.

29:42 Now, my hope is they're still also using IVOs in those places to be like, hey, we need some extra work to go in and do X, Y, and Z. But the ultimate problem of how we get guardrails in place fast enough for these frontier models, I to date, until we settle the the science more, it's gonna outpace government if they if they try to do it on their own.

30:04 And so it does need to be in partnership with these third parties uh in order to get there.

30:08 Got it. Okay. The the next thing that you said was duty of care, which I recognize as a legal term of art. Tell me why duty of care is a thing that a company is gonna care about and and why IVOs might be useful in giving them that.

30:24 So our our initial thesis on this, which we don't give up on either, is there is a lot of good that can come from tort law. And for those that don't know, tort law is essentially like when you get hurt and you sue somebody and you go to to go be made whole by being paid a certain amount of money. That's a tort.

30:42 That there is a lot of built-in liability in this tort system uh that does point corporations to do the right thing. Um there's some problems with tort law. It is long, you need to have a lot of money and about 10 years to wait to get your payoff. It's super intensive with fact finding and it creates weird incentives within corporations sometimes to just not want to know some things because if they don't know it, then they might not be liable, even if it's it's creating that harm.

31:13 And we do see IVOs as a as a way of using the sort of potential energy of that tort system and saying, what if there are a way to prove that you were being as reasonable as you could possibly be right now? We've moved off a little bit. I think our initial take was like maybe that is uh an affirmative defense.

31:31 We've really moved to the place where it's probably just enough if that's evidence of reasonable care at this point. Yeah. Nonetheless, I think could could still drive the system quite a bit, even if it's just this kind of evidentiary proof.

31:44 Got it. Okay, so basically, companies are gonna want to hire IVOs to make their products and services better, right? They'll have a reputation for quality. Maybe they'll have a seal of approval they can show to the customer. But also it gives them evidence in the event of a future lawsuit that maybe everything didn't go perfectly, but it's not like they were reckless and ignoring their duty of care.

32:13 They had stuff in place, which I think is is uh helpful clarified information. One last thing before we get to the legislation part of the story. Um who gets to be an IVO? And how do you know if they're uh a good one? Like right now, you're seeing a host of uh new NGOs, some of them founded by alumni from the labs, like uh Miles Brundage, who used to be on the policy team and I think also the safety team at OpenAI, has started a Viri.

32:43 There's a basket of these other types of NGOs. Even the traditional auditing firms like an Accenture or like an Ernst Young, which do an awful lot of stuff on uh ensuring compliance with Sarbanes Oxley and financial models, are now uh building up portfolios of capabilities designed to help companies understand what's going on with their AI models.

33:06 And um, so I'm curious, you know, do you have a strong sense of who should be an IVO and how you would know if they would be good at it organizationally?

33:17 So we do we want it to be a race. I have some personal feelings with it, which I'll just say here, which is like I do hope if the big guys get involved that they understand that this is not a check the box uh operation, that that the the big guys meaning like the the big accounting firms, like a person young or an auditing firm.

33:37 Yeah.

33:37 Yeah. You know, I think what they're used to in some other contexts is if you can prove that you did everything in a process, then that was what was reasonable. And that's AI is not deterministic like that. And also I have some real questions in the long term if that actually created, you know, if that ended ended up being what I would put in the world of compliance rather than safety in so many ways.

34:01 And so I do have in my mind that uh we do need to be gathering the milestones of the world and this kind of very limited group of experts and figuring out how to how to uh build resources around those people and and get more of those types of people to be a part of an IVO system. Now that doesn't mean that they can't marry into some things, right?

34:24 I don't see a reason why there can't be relationships between uh groups that have forever put things on accounting paper and and legal paper and understand some more of the like uh professional side of how this is done with some of the experts who deeply understand this AI world in a way that is important for outcomes.

34:44 The most important part on the government side is have you been crystal clear with an IVO what a good outcome is and what a bad outcome is? Uh and are you prepared to take away their IVO status? Uh and and I do think that it's gonna be important that we put some hides on the wall there, that there's some IVOs that don't continue to be IVOs because they failed their independence test, they failed their expertise test, or they failed to actually show that they lead to a safer world.

35:13 Uh who audits the auditors or who watches the watchmen. Yeah. And honestly, that is why government has to build capacity itself, right? Like, that is not an easy question to ask. If you leave that to like three compliance people in a basement, they're not going to be able to do that work. You're going to need expertise inside government itself to say, like, you guys really like you saw no risk in this system for biological attacks.

35:36 And then it quickly broke out and it quickly created a novel virus, or there was a near miss, or even like here was a leading indicator that everybody knows is problematic, and that we told you that is problematic. And that leading indicator was everywhere over here. And you guys didn't spot that, and you are to no longer be an IBO.

35:54 Yeah, and I think some of these things have a bad reputation because of really, really high profile crises, right? Like the accounting firm Arthur Anderson was Enron's auditor, and Enron, it turned out, was massively cooking its books, and Arthur Anderson was sort of helping them get away with that. Um, the 2008 financial crisis, the financial rating agencies that rated mortgage bonds had really perverse incentives and were basically just stamping anything and everything AAA.

36:24 So it's easy to look at those examples and be like, oh, you know, what is an independent raider really going to accomplish, or an independent auditor going to accomplish in the story? And you have to remember what life was like before that ecosystem existed, right? Yeah. It's like, oh, we had a financial crisis in 2008.

36:40 In in 1898, there was like a financial crisis every 15 minutes. Um, and like a company was cooking its books and uh defrauding shareholders or customers like every 15 minutes. Like we've got like this is an ecosystem that has a really good resume, which is not to say it's a perfect scorecard, um, but this is a a box of tools that when it goes right, it really, really helps.

37:05 Yeah.

37:05 I mean, we get when we get skepticism about IVOs, it is the from from the failure modes. And we and we like we're like we're also very scared that this will have failure modes. Every governance system has failure modes. And and you just have to recognize like governance is hard to implement and you have to be engaged in the implementation, and then it's iterative.

37:27 And and you know, even if you get it right for 30 years in a row, you you know, and it just takes a few people to be asleep at the wheel some places for it to to for a lot of things to suddenly fail. And so you do have to like continue to say, what are the redundancies that are going to help us with the failure modes?

37:46 Like what are what are dynamics? Like, have we accidentally created a duopoly and the duopoly now behaves very poorly because it knows it's a duopoly? I mean, there's there's a number of things that we won't know until we're in it, and then we're gonna have to change courses.

38:01 Yeah. So uh last thing I want to say on this is like oftentimes in Washington, DC, I hear the problem defined as trust. You know, how do we get people to trust AI? And I'm just like, what is what is wrong with you? Like that is not a good definition of the problem. We want to earn and deserve trust, not merely to have trust.

38:25 Common artists get to have trust and they're bad people. Um and I think IVOs is really about how you have trust, but also how you earn and deserve trust, which I like. So, okay, now let's pivot to the legislation because last time we talked, you were an ideas shop. Now you've got some stuff that is in law and going to go into effect in the not too distant future.

38:48 So let's start with uh let's go to California. SB8 uh SB 813 was a bill. Uh Governor Gavin Newsom has signed it. Connecticut has also passed IVO legislation, and there's a bipartisan federal bill proposal. So what's out there? Is it a perfect reflection of your vision? Are you feeling great about these pieces of legislation?

39:09 When are they going to go into effect? When might we start seeing some results?

39:13 Um so I would say like if Fathom were a normal advocacy group dealing with a normal issue, I would just say like we could probably take the next four years off because like mission. Yeah, we did it, you guys. Like we got it, we got it. But you know, a little bit of our success is just that that the issue, like we did build a thing about an issue, and the issue has now popped up and and it it is starting to to to move very fast.

39:38 And so you know, if it wasn't moving faster, I would say like, yeah, let's sit in this uh implementation zone of SBA 13 for a while and and you know um uh Frontier Act is at the federal level is uh is a great act uh for us to think about and contemplate, and maybe in three congresses it'll pass. That's not the world we're in.

39:57 Um this is you guys are feeling the urgency. We we are feeling the urgency. The and the executive order from from Newsom really felt the urgency, right? Like we went.

40:07 Just uh just the other week, right?

40:09 Yeah, yeah, where he was essentially like, well, we have the IVOs now, and the IVOs we now need to figure out, and they they've they've put our our very own Gillian Hadfield in in one of the the seats of let's get uh uh Gillian was a senior advisor for us, professor uh uh at Johns Hopkins, and and the the um the the kernel of which of all of this has grown is from her regulatory markets paper, and and she has guided us every step of this way.

40:35 She is um helping three others write recommendations about how do we make this mandatory uh for catastrophic risk, for um some of the larger issues that that can come up uh with frontier AI. Um and I do think it's time for that conversation. Uh and I think if the federal government doesn't want to have it, it makes sense to have it in in California.

40:55 Um and uh And so I think the key the key thing here is uh Governor Newsom's executive order accelerated the implementation of the legislation, but the basis of the legislation is still, correct me if I'm wrong here, voluntary.

41:10 Correct. Correct. So yes. He both said like 813 sets up the right muscle. Let's set up that right muscle much faster than we were gonna set it up, uh, because it was gonna take a year and a half, and instead he's like, get it ready in six months. Uh and let's decide how we're gonna use that muscle to actually have a man a mandatory regulatory uh system.

41:30 Yeah.

41:30 Um and I think one of the other things that Governor Newsom did that I thought was really interesting is he sort of built some synergies between SB 813, which I think you can fairly call like the IVO bill, and SB53, which is California's landmark, uh what I would call AI safety transparency law. Um can you talk about like what the linkages are between what's going on there and how it's gonna work in California?

41:56 Yeah. So I mean, I think the basis for all AI governance law right now is transparency and then checking checking the homework of the labs. I mean, like if you were really gonna break it down to talking points. Um and uh and Gavin Newsome has said, like, hey, like one of the faults of 53, which is a self-reporting uh system of uh, you know, where do you see the potential for catastrophic risk and where do you do incident reporting, is that we're just letting the labs do it and define it.

42:26 And where could we actually bring in um IVOs uh to do that actual reporting for them, to actually be the ones to say, um, here's what's happening, and we uh uh and we see this catastrophic risk or we don't see this catastrophic risk, and it's suddenly the IBO's judgment call and not the lab's judgment call on on this uh the this transparency report.

42:50 I see a lot of um good potential there. Um uh and then uh and I I do also have to say that's not that far from what the Frontier Act at the federal level is as well. Like if you actually go back and look at the frontier level, it really does take a lot of SB 53 and and the RAISE Act and the transparency bills in general, and say that's the first half of the bill, and the second half of the bill is how is that work being checked, augmented, and risk assessed by this IVO system um uh as well.

43:20 And so I do think we are converging on uh at the at the government level, we are converging on what people want to see out of an AI governance system.

43:30 Okay, so the the bill will be effective January 1st, 2027. The IVO criteria deadline, correct me if I'm wrong, has been accelerated from uh January 2028, which was the original deadline, to May 2027. Um I don't mean to accuse you of not working hard because I know that you're working very hard, but is this all even as fast as your guys going?

43:55 Is this going to be too slow?

43:59unknown Yes.

44:00 Yes, maybe. I mean, we don't know. I appreciate the honesty. Yeah. Yeah. We um there there's a uh a great um uh uh substuck out um by Anton Like to um who I will also say is uh now a a senior advisor to Fathom as well, uh called Send Them In, which is kind of like it's gonna be great that that uh that government is now suddenly interested in growing this muscle.

44:24 We also need to start sending them in. And uh uh by them I think it means third parties that that can start to shed some light on what what happens. We are spending a good amount of our time right now saying, like, how do we make sure that's going as well as possible? Yeah. Not just for the lessons learned in the future of governance, but also like that is the beginning of the muscle building.

44:45 Muscle building is actually happening right now because people are, you know, open AI, anthropic are actively negotiating with these third parties, actively getting into arguments with these third parties, actively um finding hiccups and and but also discovering new things, right? If you if you go side by side between what meter uh reported out of the hug and face incident incident versus what open AI did, meters was uh a wealth of new information for a much deeper understanding.

45:16 And that's an early win of this third-party system, right? Uh, and we need to understand, start taking lessons learned from that and really start baking that into the when when government gets more directly involved as well.

45:30 Yep. Um, now we've been talking about California because their actions on this story have very much been in the news lately, but I don't want to give short shrift to uh Connecticut, uh, which actually did this first. Um, is it is it kind of a one-for-one, pretty similar approach to the implementation?

45:47 Are there meaningful differences between the two laws that you see that that are worth commenting on?

45:52 So Connecticut is is a pilot program for five of these, which I think will be really interesting to see which five kind of raise the top. And I do think that there's this opportunity to show interoperability between two states with it. And I and that is what would be the most interesting to me if it comes of it.

46:09 They're both, I think, have the leeway, if not the direct order, to kind of make sure that they are reaching out to each other uh on this. Um uh but that would be important to us is is how do we start creating that uh interoperability moment. Um the other thing I think interesting about Connecticut is that really does allow them to go deeper in five than um uh rather than like come one, come all, which is really the system in in California, is you get to apply and then we we work our way through these.

46:44 Um uh we are involved in the rulemaking on uh in Connecticut. We're part of the rulemaking team there. And I would say like what we'll be interested in knowing is like how do we prioritize what five we start to look at at the beginning and and where that goes.

46:59 Yeah. And uh I I guess it's also worth pointing out that um the connection to liability is explicit in the Connecticut uh bill. Sorry, I should say Connecticut law, because verified companies do officially get evidentiary support in private civil suits. Um, and that's I don't believe explicitly spelled out in the California.

47:19 There actually is a line on that in the California bill um that uh it is indicative of, but not uh but does not actually prove um uh some some sort of reasonable care. It's at the right at the end of 813. Got it.

47:32 A nice little tack on at the end. Okay. So now let's um shift gears here. And uh let me ask a question that I ask uh with total kindness. Um we've gotten this pretty remarkable set of safety failures that are collectively called the Hugging Face incident, although OpenAI has apparently notified a hundred different parties of uh these kinds of autonomous hacking issues.

48:04 So it's certainly not just Hugging Face, although we've sort of been collectively calling this summer of AI safety failures and now fall of AI safety failures the hugging face incident. And I just want to ask like, you you mentioned that meter did a good job writing the post-mortem of that. But do you feel strongly, one way or the other, that if IVOs, you know, if if people had implemented your idea the day Fathom showed up in 2023, 2024 timeframe, would something like that have prevented hugging face?

48:35 Would it will it prevent the next round of these kinds of incidents?

48:39 I I I'll be confident in the first one and less confident in the second one. I will say conf confidently in the first one. I do believe they would have they would have stopped it. Um that that um one of the cleanest places that IBOs can kind of come into existence the fastest is cybersecurity. There's been cybersecurity groups with uh and and not to like not to belabor the the the finding of it, but like there were some things that could have happened that would have uh stopped stopped AI from from actually reaching the open internet that that from my understanding and talking to a number of cybersecurity experts, they're like that would have been best practices to to have that implemented at

49:19 that point and and and and brought in. And so I do feel fairly confident saying like if somebody was watching over their shoulder that was that was like we would get in trouble if there was an escape, and so you better be doing this, um uh it would have caused a greater level of security. And even when I hear open AI employers talk about it in blog posts and whatnot, where they're like, you know, it was kind of a lack of our imagination about what these things could potentially do, which I honestly felt to be like a little bit of a weird blog post because I was like, if you guys aren't imagining this thing, like where else are they like like that?

49:52 Yeah, who's supposed to imagine this if not you? That's right. Right.

49:54 Um that that an extra like having to show your homework to uh to outside experts would have would have actually I will say, and and this is uh not so much uh about IVOs, but I I just want to say it. Um my credit to you know the open AI safety folks of like two years ago, my respect for them has gone up because you know, they were writing all of these re reports and guidelines and safety frameworks, you know, which at the time were not getting especially positive press because most of them said, you know, if this happens, we would deem it critical.

50:32 If it was critical, we would impose such and such safeguards. And all of that felt like to a lot of people in Washington, DC, just felt like marketing materials two years ago when they were writing it. And then today, when it's like, well, you know what the critical cybersecurity institute was? Autonomous escape, autonomous hacking, collaborative behaviors.

50:52 And like when those boxes started being checked, it really did drive the behavior of the companies in a positive way, which I thought um was uh was a really deserving of praise for the work of those open AI safety folks from several years ago.

51:08 I I and I'll say to to like bring on the a little bit more praise is I've seen an actual tonal shift within the company too, right? Where like maybe that was just their safety researchers at the beginning. That is now permeated throughout the culture.

51:21 Uh yeah, and I mean they they were opposing some of this regulatory legislation in California, and they've literally switched in the past three or so months. Um, and they're now in favor of it.

51:31 Um so the the second part of your question from before is will it stop the next one? Here's here is like there is an actual tech debt between capabilities and safety tech right now. And and I cannot with that until we like really start to put resources into it, um, and we start to see how it catches up or doesn't catch up, we can't say where the safety tech when safety tech would actually set like reach parity with capabilities.

52:00 Uh and um that is that's an unknown uh that is a known unknown in in this in this whole thing. I will say something something that's hardened to me is like I I think there's literally hundreds of millions of dollars in philanthropic and venture capital money coming online to create IBOs right now. I mean that like that that science is gonna catch up um uh uh quickly.

52:23 It needs to reach the billions, which is a crazy number to say, but you know, compute's expensive and yeah, wow.

52:28 And yeah, so okay. So um now I want to shift. We started with politics and polling. I want to shift back to politics and and bring in you know some noteworthy statements from folks affiliated with this administration and this administration's policy on AI. And uh let's start here. Okay, so the 2025 version of SB 813 that Fathom sponsored at the time offered certified developers a legal defense.

53:01 The administration, the Trump administration, is now attacking a version of that idea. So here's what Treasury Secretary Scott Bessent said. I believe this was before uh the House Financial Services Committee. Quote The one thing we should not do is give AI labs a blank check on liability. The best way to guarantee safety is that the creators are liable for what they build and generate.

53:26 And they are saying that we would all like to slow down, but please give us a waiver on liability, which should not be done. And I would encourage everyone in this committee and in both houses not to consider that. And then uh David Sachs, the former White House AI and crypto czar, he said on X uh last month, quote, stop pretending you need anyone else's permission.

53:50 Stop pretending antitrust law has to be suspended so that you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. You face massive product liability exposure if your products enable a truly damaging cyber attack. End quote. So you've got two of the most influential voices in this administration making a version of the same argument, which is that tort liability is the safety regime, and that any certification that softens liability is actually not going to make us safer.

54:26 It's really just a gift to the AI labs. So, you know, Fathom has made arguments that has proposed bills that intersect with the liability issue. We've talked about some of them. What do you think about the remarks that we just heard from those two uh folks in the administration?

54:44 I I would juxtapose them against the the accords that the the that the president just put out, which had that part of the system that they want to have come in as third-party evaluators that then would be subject to reporting directly to the board, like even without being written into law, would then definitely serve as an evidentiary uh it would certainly come up in the suit, right?

55:05 It would it would serve as evidentiary proof. I read some of that as like don't accept them from home. Like don't do what we did, uh don't have a full liability shield. Don't do what we did with social media, right? Don't don't uh have a section 230. Um and that I absolutely agree with. On the on the point of don't have a way to prove reasonable care until like until after a critical incident and like a liability group, I would probably have some pushback on that, where it is like forever our systems have been built on both things, right?

55:37 That there is both your day in court and some things you can do ahead of time to get uh permission instead of forgiveness, right? Figuring out the right balance of that is one of the many difficult policy questions. And I would say original Senate Bill 813, we probably leaned a little bit too hard into the permission framework, uh, and we've worked our way back to evidence.

56:00 At that time, we thought we needed it in order to jumpstart the IBO system, and now we don't think we need it in order to jumpstart the IBO system.

56:07 And and even if it's not a you know shield from liability, you can still get other really meaningful financial incentives, like, for example, lowered insurance rates. Right. And not just for the AI developers, but for their customers who are using this technology.

56:20 Although I would say like the way that all comes back to each other is you get lowered insurance rates because somewhere at the end of the line you think there's less liability risk, right? There's also like the point of like, it's no good if it doesn't actually lead to safety. But if it does lead to safety and you are trading true safety mechanisms for decreased liability, that's a good trade-off.

56:40 Uh so then the question becomes like, okay, how much do you trust this that the system you've created actually is creating good outcomes? Which I do think like, you know, this is a this is a a dial and not a one-zero. Uh that we push it a little bit further one way if it's working well, and we push it another way if it's not.

56:59 Yeah. Um, okay, uh, this is phenomenal. So on tap for Fathom in the next year, um, you've got the the Frontier Act, which is being co-led by Republican Jay Obernalty of California and uh Democrat Lori Trahan of Massachusetts. What do you think are its prospects for passage in the coming year? I mean, what do you think we're gonna see writ large for AI safety and governance, AI regulation over the next year?

57:30 And what role are IBOs and Fathom gonna play in it?

57:33 If you'd asked me four months ago, well, I I forget when introduction was, probably closer to two months ago, I honestly would have been like, hey, this is like a really good way to start talking about these things, but I don't yet see a path. I now actually would be like, there is greater than um uh single digit percentage chances that that there's uh an AI bill next year that that makes it all the way through.

57:53 I've just seen enough tone shift. I've seen it like I've seen enough tone shift from enough um people who were were pretty hardcore, no way ever. It will still have to be bipartisan. So like the the like the shout out I want to give to the universe is um uh the place where this always goes to die is the 60th vote in the Senate.

58:12 Um uh and we have to remember that we need those 60 votes in order in order to get there. But I do think like a frontier bill as as the sort of framing for that gets at a reasonable package that could pass. So, you know, do I think it would be great if some of the parts of preemption are softened? Sure.

58:30 Do I think that it would be great if instead of right now there's this part in it where you can only you can only hit the big red stop button if you are if there's imminent catastrophic harm, and that probably shouldn't be that way. There probably should be a way to to slow down the system before right before the moment, right before everybody dies.

58:47 Uh so there's things in the in that bill that probably need to change um that I think would change in a Democratic Congress. But I also think uh we should remember that the next two years are gonna be really, really important for AI governance. And we should remember that that still takes the president signing it and the 60th vote in in the Senate.

59:07 And that's the place where I still get like a little bit queasy in my stomach of whether or not it will get across the finish line.

59:13 Yeah, I mean, I mean, uh it's hard, it's hard to say you have a better than 50% chance just because this is the United States Congress we're talking about. Does anything have a greater than 50% chance? But I will say, um, when I talk to folks on Capitol Hill, uh there are there are bills that are introduced as uh this is this is gonna come off harsher than I really mean it, but there are bills that are introduced almost the same way you would publish a press release, right?

59:42 Like you're you're issuing a bill to make a point uh and to to generate a a news cycle. I mean, what Bernie Sanders, former head of communication, even admitted, right, that that's that that explains a lot of the bills that Senator Sanders introduces. He's now introduced bills, you know, to ban the building of data centers or to ban the development of artificial general intelligence.

1:00:03 I don't think he expects those to pass. I think there's a lot of folks in Congress who really wonder if this is the bill that we should pass. And they're either curious or they're supportive. And, you know, one of the things that when you're talking about an industry such as this one, you have to say future developments are definitely going to shape the environment in which legislation like this is considered, right?

1:00:30 In the same way that the hugging base incident accelerated the implementation of the California bill, uh a second round of major AI safety failures would almost certainly really shift the conversation again. And you could go from having a 10 to 20% chance to having a 60 to 70% chance. And that's the the virtue of doing the hard work of coming up with shovel ready ready policy.

1:00:56 You nailed it. You nailed exactly how we think about it. And so in that way, we're in the perverse world where we actually hope that this is has a low percentage chance of passing.

1:01:05 Because you hope humanity has has a good long time.

1:01:08 We don't have another incident, but that we are preparing for this to to fill a need. Uh cool.

1:01:13 Well, Andrew Friedman, uh, really enjoyed the conversation. And thank you so much for coming on the AI Power Podcast. We'll have to have you back on when the next round of polling comes out.

1:01:22 I will be here as often as you want me, Greg. I continue to think that you are the best podcaster in the business. So uh happy to happy to be back on.

1:01:28 All right. Thanks so much.

1:01:30unknown Great.