Google takes down the internet!

0:00 Hello everyone and welcome to stand up. I have to step away for a second. Shut up. Oh my god Dude every crime moment, dude. Yep. Hey. Hi. Hey everybody There is a very high chance. I have no idea what we're talking about I was a hundred percent chance. I have no idea the Google cloud service outage last week Okay, so you didn't really a security thing, but that's okay. Did you wait you made a video on this?

0:28 What are you? I did and it doesn't mean it's not security. I need to be honest and there's which what yeah, it's not Security. We're talking about no one got I did make a video on it. Just throw that out. I just know Yeah, I made a little quick video on it, but I have it. I got I got all. Okay. That's all good. Let's do it. Let's go We got this. We got this time. Nothing can possibly go wrong. We're good. It's great step away. No problem We got go pee pee or something. Yeah. Okay, the reason why I got to be right back is I got to put a little bit of milk in my Coffee because just a splash of milk with cacao is really delicious. I would love a coffee right now actually. Oh, I'm on coffee so bad

1:09 Prime I know that you're you're itching right now to do your signature opening to the extent that it exists for the podcast I the single greatest opener of all time for the standup would love to start the opening. Hey today We have a very special episode of the standup. We replaced TJ with in fact a smarter version of TJ Low-level learnings first appearance on the standup security expert probably Bellotro expert as well as always We have Casey Moritori and trash dev on here today. We're going to be talking about a very large security incident and Low-level learning we brought him on due to all of his expertise his good looks his deep knowledge of C and Null pointers and everything and he's going to walk us through what's kind of happening

1:59 We're going to talk about maybe the state of security and who knows where this is going to go So low-level want you kick us off on this one. Let's do it man. Yeah, so I mean it all goes back to 1972 when C the programming language was invented and then 50 or so years later we're still using it, which is kind of crazy, but yeah, man last week Google kind of took the internet offline for a whole that time out time out Yeah, yeah, yeah, yeah, you started at night You started like 50 years ago and then went right up to last week like what happened in between Not not a ton honestly like not a ton like a couple new c standards came out and then now here we are So that's about where we're born. Yeah, some people were born

2:38 But yeah, so Google took the internet out man, and it's kind of crazy how they did it Uh, it was like Thursday at like 1352 roughly Eastern time in the U.S. I have to convert that in my head One PM Eastern time boy, you just got to deal with it 10 p.m. Pacific time, and I guess you route it to two o'clock I guess it's 52, but yeah internet went down for a little bit and no one still to this day Know exactly what happened, but Google did put out a report that talks about what they did The end result or the end reason was that Google published code that did not account for null pointers And when a policy change went out that contained a null pointer in the policy change It crashed the management plane of the Google Cloud period full stop. That's what happened

3:23 Absolutely, and the last thing crazy stuff classic Google move and the irony in all of this is that Google is a company that runs the OSS fuzz project which is open source software fuzzing So literally what they have is a like data center full of Machines that fuzz open source software to find memory corruption vulnerabilities in them, right?

3:42 This is a project Google has been sponsoring for a very long time So the the palpable irony in Google releasing software Internally to their own management plane that had a memory issue and not like a like a heap overflow or race condition Something a little harder to detect something as simple that you could like lint with like a and a failing to check a null pointer Crashing the internet is just some of the craziest news. I've seen it a very long time. Okay. Can you can you rewind a little bit?

4:09 You said two things that I think probably need a bit of explaining one Maybe give like a quick overview of what a memory fuzzer is and then too You said that nobody knows what happened, but Google released a port are you saying that Google knows what has happened They're just not telling people what has happened Front down. What happened? I don't know. Did you post on a Friday?

4:27 Never are we going to figure this out? How are we going to figure this out? That is your proudstown? Did you guys need the wheel? You okay? I'm getting sick. I'm just for the couple hours. We're back to right off the plugins Don't guess where your issues are you can see exactly where they are happening with century Get all the context you need to debug any problem because code breaks so fix it faster with century Operationally internal to Google we do not know how that code got out right like was the code pushed through cicd And like it failed the test and they just said roll with it Was the code vibe coded and no one really checked like there's a lot of like weird if and if ends I could be happening with that code that causes to get published in the state that it was that's a little weird

5:22 That's what I mean by like no one knows what happened We know that there is a null pointer. We know that that was the reason why it crashed But how that code got released is still kind of like ambiguous and then case you and I were talking previously about like when crowd strike had their incident Right They released like a 50 page report that said like hey, here is exactly what happened Here is why the code went out, you know, sorry It won't happen again. Yeah, yeah, yeah, a Google has not done that yet And using Lee these two exploits were I've been exploits the right term sorry these two crap like they're just crashes like nobody Exploited anything right right But they're actually kind of similar the crowd strike one and the Google one were both like we have some config stuff that can happen

6:01 Essentially like there's some kind of like code that's being driven off of like some data that goes into the code that tells it like what it should do Right And in the crowd strike case that was the problem is that you know that the in production was the first time it got fed The correct data like to make the bad case happen And the same was true of Google apparently it's like basically this code You will only hit its null pointer case I mean trying to read between the lines of the thing that they published because again They didn't show the code like crowd strike did But it only gets to the null pointer case apparently if you pass it a certain configuration for the quota system Like this it was in the like thing that's trying to check

6:40 Quotas or this sort of thing So It's the same exact kind of pathologies like there's a type of data that if you fed it to this thing It doesn't do the right like checking of that data to make sure that it won't go into some bad code path And it off you go and so they're they're kind of actually somewhat similar at least from what Google actually said Which again, they didn't include the code so it's like who knows they didn't even say what language it was incorrect They didn't they said null pointer but that could be C C++ whatever right or go Go as null pointers. I think you're right. Yeah, yes, they do so famously hold on time out Did we explain what a memory fuzzer is? Yeah, I can talk about that. Okay, so basically I'm

7:20 Basically picture. It's like a unit test where you're trying to get as much code coverage as possible But instead of like traversing down paths with like the same data shape You're literally mutating the data to try to find edge cases in your code that are not accounted for in testing So like If I have a an HTTP fuzzer my initial corpus my first C is going to be HTTP you know like a get request right and then what the fuzzer will do is it'll pick a random bite in that packet It'll mutate it by like flipping a bit It'll like make an integer a negative integer It'll do some mutation and then it'll run that request on the service and see hey does that input generate new coverage If it does okay, that is a new piece we have to add to the corpus to future mutate to find new code paths

8:05 And you do that trying to enumerate the entire code base from a coverage perspective And so what a memory fuzzer would have done if they had fuzzer software and again like this is the the the science behind getting coverage in fuzzing is like cutting edge technology right now No one has it like figured out but Google is very good at it A fuzzer would have very quickly found like oh if this field is zero it crashes a program right?

8:27 So it kind of implies the software was not fuzzed at least long at all To Casey's point this code had actually been public or not public It had been released since like May of this year like May 29th I think and because they had never pushed a policy change that exercises code path It just never was seen to crash So either they didn't test it they didn't fuzz it Nothing really was done to like try to exercise this code until it got hit on in on June 12th And it sounded like too that the fuzzing would not have had to have been very complicated because The the way they made it sound anyway in the very terse description was that like the kind of thing that would crash This was just something with some missing fields they used to they just use the phrase missing fields

9:08 So you know a basic fuz tester would probably try a bunch of different field combinations with some missing Okay, like that's not a weird thing to have fuz and apparently that was not done I guess but again because there's so little information who knows maybe it was a very strange like really unusual Combination of missing fields that no one thought to check who knows right? Yeah, what I would imagine is again all speculation I don't work at Google. I have no idea It is likely they're passing around either a JSON blob or a YAML file and then like if you use some YAML object That exists but a subfield in that YAML object does not exist that will populate an old pointer that then gets read And the problem with that is like fuzzing fuzzing binary data is very easy because like I said you flip a bit

9:53 You make an integer sign something like that fuzzing Data with a sane grammar like JavaScript for example or YAML much more complicated So it just it the fact that it was not fuzed implies that it was some like human readable grammar and like a A config spec That just didn't get tested which is again for Google. It's just crazy, but also I could totally see why that would happen operationally So I have a question. Yeah, what's up?

10:21 So I don't know much about the outage because I was out of conference So I didn't really see the internet go out if it actually did at all So when you say that crash the internet to me that just sounds like a really bold claim like Like like yeah, yeah Like what is like the impact here and like What was the collateral like what happened exactly? Yeah, so basically went when this outage happened what really ended up occurring was that the binary that was responsible for doing quota and authority checks or like authorization checks on requests like crashed in all regions of the world, right?

10:54 So basically any call to a Google cloud platform API would just 503 like you could not do anything So if you're like my course website not a plug this like real life What's the course website again just roll-level academy if you want to learn to see Ross whatever Yeah, yeah, but I But there's my saying this because I Depend I depend on Google OAuth right and so every time my site receives a 503 from an API I get an email and so during this outage because the OAuth provider was returning 503s I got like 120 emails in 30 minutes being like you know the internet's down. So the issue was every Like basically every site on the internet now Depends on Google in some form or fashion through OAuth OAuth

11:40 Through the actual like compute providers a bunch of other stuff that I don't use or don't care to use But basically Because Google and its entirety went down it kind of like brought the internet down with it Which is very like counterintuitive to like the design of the internet It's meant to be like a decentralized mesh of nodes that if one goes down routing exists and you can kind of route around it But if it kind of just goes to show that we have this really weird dependence on like single points of failure and modern cloud architecture Which is kind of scary I think from uh You know A cyber attack perspective, right? It's kind of like how when dine went down or didn't have you want to pronounce it in 2016

12:13 Like again DNS core protocol to the internet There was like one major provider that went down because of a botnet and when that went down it took the whole internet down with it So cyberdine did you say they're just dying? Yes, sorry cyber attack the provider is called dine. Um, I don't know if it's dine or it's a terminator joke He's going sky net on us. Sorry. Sorry. I'm a little too young for terminator. I think I'm 30 Yeah, shut up. You're not tm for term. You're already dead. What's the problem is when the first terminator come out?

12:43 78. I think oh is it that old? No, we already 81 or two 80s Terminator one. No, I don't think so. I think I think it's late 70. No, it wasn't terminator. It's the best time in 84 here right as I was saying Yeah, but the problem is you can name any movie I probably haven't seen That's okay Lord of the Rings I've okay those have seen But I'm a nerd any other movie fine. Yeah, really just any other movie Do you do this again?

13:17 I'm not worth the only two other than Anyways, I guess we could just move on mr. I've never seen movies except for all the ones we name Exactly of the two. Yeah before before Maybe we should just stay off topic. I was gonna bring us back on but that this is kind of good name a third movie So I wouldn't name a third movie. I I won't with gladiator. I have not seen that yeah, that's a little more saving private Ryan I've seen half gladiator is not rare. Okay, well Is one of the best movies ever everyone stop how did you see half of saving?

13:55 Can I tell you why? But I was growing up big paintball kid. We love to play paintball We go drive an hour south to go play paintball called cousins paintball and not Tom's River New Jersey And every time we went we would hype ourselves up by watching the Normandy invasion scene We would get like an hour into the movie and then we would be done That is dark That's what we would do we do we were like 12-year-old kids man We had to get hyped up somehow We were going to war with paintball man. Yeah, we were teenagers. So I mean like give me a break But yeah, so that's why I've only seen half of it because we would get into the Normandy invasion scene Get like an hour into the movie and then we over and yeah, we'd be there

14:38 She's like, I still don't know what happened in World War two. Who was I mean did we win? Shit, I have no idea. It looked pretty bad from that Yeah, no spoilers. I'm apparently like up on hire. We had like a big bomb or something Yeah, when did that even happen? Who knows? I'm confusing very confusing. All right. Well, let's get back on topic here. Okay, so so philosophically Are you saying that all those times? I've been told that rolling my own off is for losers and people who want security issues May not have been correct. Is that true? Is that what you're saying low level learning chief expertise of security?

15:15 I mean chief expertise in cloud incorrect. No, but yeah, I don't It's tough, right? It's just everything you do anywhere including in programming including in like Like system architecture is a choice in risk and what is risk? It's the probability that a thing will occur and if it occurs what happens right how bad is it?

15:35 The probability that Google Cloud goes down all the time is fairly low So not impossible, but like you know, it could happen So I don't think it's bad if you use Google off There's nothing wrong with that But you have to consider that in your calculus like well, you know How mission critical is my software if if Google goes down, which it definitely could Are people gonna get hurt kind of thing like maybe medical providers shouldn't be using cloud auth for further stuff If they if it's no a complete necessity that they get into it So I don't think it's bad to roll your own off. I just think you have to do the math on what makes sense for your application You know It's kind of rough to I think like at least from a

16:12 From a user's perspective or I mean for for when I say user I mean someone who uses like cloud services It's a bit tough to because one of the things that seemed to get exposed by this particular crash At least when I was kind of looking at at people's reports on it Was that a lot of people didn't seem to know or Or Maybe could not have known That they were relying on someone who was relying on Google. So you might I could even imagine somebody was like Oh, don't worry. We we like use two different Storage providers or two different author providers not realizing that actually they will both go down if Google's quota server goes down Right, and so one of the things that kind of occurred to me during this crash not being a security professional

16:57 So I don't really have to think about this for my job But it was just was kind of floating around my head was I was like You know, it probably should be a requirement somewhere for these kinds of services that you get the diagram Where it's like what are all the things that you actually call out to so that I can look and if I want to do a redundant thing I know that I need something where the Venn diagrams don't overlap because if at the end of the day all these people call AWS Then really I have never done this at all because as soon as AWS goes down all my stuff goes down or whatever And I realized no one's probably thinking that way like like people like the web is not really about Uptime it doesn't seem but like let's suppose you were it would be interesting to be able to get that so that you could just kind of know

17:39 Okay, if I use this and this they're fairly disparate whereas if I use this and this they share a bunch of services So there's probably no point because I'm not really getting any redundancy does that makes sense what I'm saying It makes sense, but I do I do want to throw something in here really quickly before an actual expert speaks Is that case it sounds like you're you're getting dangerously close to us drawing umel which uh, just general speaking here at the startup We're very against the umel so that already sounds like hey, we just need like a universal language for modeling all the services The dependencies you want greedy booch on here. You got some booch. You got to boot you know, I don't want booch on here

18:13 Booch is coming on. I'm gonna call him up. I don't know him, but uh, you know, call him up ran believe on. Hey, Grady Let's go in on man. You don't know me, but I got this podcast that no one wants you on apparently could you come on? Yeah, okay Well, no, I'm not talking about a UML diagram But I guess I am sort of scarily talking about maybe like what did they call this sister systems architect diagram What are they called that little thing where they draw a bunch of little cloud shapes and then they write a name in it and then there's arrows I mean, I don't actually need that because I just need the list right I just need to know like what main services, you know, are you depending on or something like that

18:51 But I do think that would be interesting to know just from a from an architecture standpoint because it's like If you think if you're thinking of this service as just an endpoint you connect to and so you're just thinking what is the chances that that goes down You may not realize that really that and some other thing they will go down at the same time if this other third service They both depend on goes down and that's like to the point A little level was just saying about like calculating that risk That's an important part of the risk calculation, right? You have to know which things are independent because like the chance that Amazon and Google have an Outage on the same day is pretty low But if actually they're secretly both using each other's stuff

19:29 It's very high and you need to know the difference between those two things. That's all Yeah, I mean that you saw that when like as a result of this Google had an outage But it also caused cloud flare services. Yes, not as fast as how did that happen? What was this because I just I was I guess I was a bit shocked when cloud flare was reporting things going down Afterwards because I heard this was like a Google related thing. Yeah, and so I was very confused as to how the Other services went down Service thing they do and it was in it relied on Google cloud apparently Oh, were they actually related was the club. Yes, I believe that was the case So that's the only out I heard of was the cloud for one so I didn't realize there are kind of like one in the same thing

20:08 Which is really funny because at the conference we're out of cloud flare booth and we're like showing them the The stats like you guys are down You were literally walking up to people who are I think I think hacks or did it Your your entire service is down have a nice conference Sorry, I'm looking at the other report pack up the flare cloud flare is report when a lot more in detail to than the Google's did So it looks like yeah casey said workers as well as so what is it workers warp access gateway images Okay, workers AI turn style auto rag. I don't know if it is in Zerad. So there's like a lot A ton of services are downstream dependent on Google at what you would call you know a Csp right a cloud service provider, which is just again to Casey's point like you can't make a sane risk calculus

20:58 If you don't know like the probability and the outcomes of things happening So like if Google is like the core call it center of gravity right of all of these things If you if it goes away, what happens? You know the because you're not exposed to that you can't make a good good call off of that I have a question for you a little level. What's up? So obviously like you say your question trash I'm gonna say my question, right? You're ready? 321 Hold on articulating articulate. Okay, so if you let's picture like a big company. Yeah, like Amazon or something Let's see Mr. Pickerman's good company. It's like Netflix. I was thinking about this question for like 10 minutes And it's just like okay, okay, okay. Let's just picture. I'm like mr. Amazon and I have a lot big company

21:46 so many teams is hard for me to find like Which team is like is actually dependent on something that could actually like Like the dependency waterfall right like how do I like if when you're consulting or have you seen this in practice like How do companies approach this like to know that you're like the single domino like if you fall We're all like fucked right like yeah, because it seems like when things like this happen It could it's clearly like not in anyone's like peripheral, right? I don't know. Yeah, I'm gonna say I don't have a good answer to this I've worked at big places and I've worked at small places and inevitably what happens a companies that get big is their knowledge management solutions began to suck very bad

22:26 And people that within what you would call like principle engineering positions No less and less about like the actual principle engineering concepts of the architecture Um, and so at the end of the day like nobody Probably knows the totality of like what you depend on and I'm not sure there is a good answer I mean, I think the real answer is like you have to have a single guy that probably knows the entire setup and have like a good place Everyone can go reference and see like kind of what case said like the UML diagram of the architecture right like hey We depend on Google here here here if it goes down we can make this this risk calculus Um, but there really isn't a good answer to this is probably why this not that it keeps happening

23:01 But like that that this is able to happen because no one's really aware It how it all kind of glues together. It's it's hard. Especially at a company as big as Amazon or Netflix potentially Um, just a big random company a random just completely Yeah, it's all like so question for prime So obviously we have like chaos monkeys cast month kind of like in the same spirit of I don't really know what it does at a little level I'll tell you what cast monkey cast Kong and cast gorilla or cast gorilla cast Kong So we have three of them and no idea Hey, I gotta tell you about Netflix that you worked there and I doubt so that So cast monkey what it would do is actually target an individual Instance in Amazon and it would kill that individual instance

23:42 And so that way it's kind of to prevent any sort of like stateful operations between two separate services Where I'm caching like I'm doing some sort of extra curricular caching and creating some sort of weird state dependency It will start just randomly killing services and that runs in production every single day as far as I know at Netflix chaos Kong Or chaos gorilla will take down an entire service. That's what we have something called blue services at Netflix So if you look into blue services. That's like how we do a lot of Default responses and things like that. So if a service goes down you're able to have some form of data coming up That's what we have a default lomo because if you can't reach GPS or map or or you know all the things that go into creating

24:22 There's like picture. There's racks. There's all all those services all together If one of them goes down then you get what is the default lomo Which is like the latest known highest ranking videos that will all come down So you'll get like stranger things right off the rip even if you've seen it and there's no new episode Just to translate this for prime using knowledge. I gained from prime stream They use the word lolo mode to mean list of list of movies. I believe I didn't know that for the longest time Yes, it's and but we now we technically are a Lolo Romo, which is a list of list of recommendation objects for movie Objects. Okay. It's a new explain that for another one I actually was the first person to implement that it was very tricky, but we got it done

24:59 Anyway, so with all that then chaos Kong, which is you will see emails about that So if you just searching your email chaos Kong it usually runs on a Wednesday and What will end up happening is that we'll actually take down an entire region So not just a specific service will actually take down us east one or us east two or west one or somewhere in EU And it will be it will be ran in production And usually it's for like one or two hours to make sure that if some region goes down Netflix continues to operate correctly So those are the three levels and so this doesn't help because the problem is is that if every single one of these services Has some sort of dependency say on Google You can't take down a region to test this because you don't take down a region

25:39 You're taking down an external service which would crash every single subservice which would take down all regions Right, so there is no overflow mechanics So very interesting thing to do a chaos monkey with which is to Kill that service for or a chaos Kong at least would be to kill that entire service for a moment and see how How does your system handle say a Google going down or a Something but then you don't also know like all these like you said like okay I would know that cloud flare was going to get taken down because Google Didn't access to a no pointer. I have an idea. Yeah. Yeah. Yeah. I have a great idea you guys can have this for free at Netflix I'm just giving this to you. I'm giving this one to you. I think he's on the same page as me. I'm curious

26:16 Okay, so here's one of the same page. What's happening? So here's what you do Because obviously you got you do have a problem because Netflix is like the the whole point of service isn't so To play movies so those three movies at home or wherever on my devices and You get in this situation Google Cloud goes down Amazon goes down whatever the whoever storing the data goes down I can't get the movie so here's what you do You get a really and I mean Blocky compressed version of baby shark. So we're packing this thing into like a megabyte, right? Okay, and every single Server node you have no matter what it's running keeps that one megabyte in memory So that if everything goes down Netflix will just stream baby shark to everyone so that at least something is playing during an outage

27:08 And then you can just be like you can mix up something about what have me like oh Sorry, you know like our recommendation service is the law on the fritz So it just decided that everyone wanted to watch baby shark and then it will seem like you just have like infinite uptime There you go. Can I tell you a fun side story this just reminding you of a side story Okay, I tried to convince our marketing wing to do something kind of similar Oh really in a similar vein so effectively they they did this once So if you look back in the day this must be 2016-2015 We actually had leaked I think the second season of house of cards early Okay, and everybody was like oh my gosh house of cars and they got like for 30 minutes

27:44 It was up and people could go watch it So we actually like had content that we shouldn't have had leaked and so then there's all these articles written All this hype built in all this and so I actually wanted a I know this is not actually related that all to your idea It's just it just reminds me of it. I actually I tried to convince Netflix to leak Shows more regularly Just for like a half an hour because then you get all these articles written everyone gets super hyped up And it's like such a sweet way to mess with people and it looks like it's an accident And so you can have this whole idea of like accidental content You can just be like oh our fault, you know, it didn't mean to do that one

28:19 And then published like a fake apology every time right? It's like making up service names like our Gorgonzola service failed And then they're just like they're they don't actually exist internally Great love Sir, you know, sorry low level you wanted to say something that was not my ridiculous maybe shark Well, it sounded like you were going down the path that I was gonna go again, but you didn't it's okay You you're done a completely different path. Yeah, I propose Chaos long you're me out. Okay. Okay. Okay is you this is actually real like what you could do is you can create like basically a firewall red button layer in between a cloud service provider And like whatever service it is. So just like you're doing like chaos dong. What are they called chaos con?

28:59 Yeah, you know chaos con attacks and that's long You can also do yeah, and then like if it's a if it's a regional cloud-based singing bee chaos con dong or Chaos, you know, if it's just for you know a small little service chaos monkey dong But all you're doing is you're red buttoning the the the cloud firewall to see like if Google goes away what happens right My rate is roughly 1200 an hour that will be 120 bucks. You can build to Netflix trash The only problem obviously the only problem with that notion is that there's all those third party one like the cloud flare one Then leaves your cloud or leaves your private network that you can control your whatever however You're controlling the traffic through your private network and you wouldn't be able to control that and so they would still remain up

29:43 Which would hide with problems? We're actually asking this is that this is like the sole problem Which is the internet is supposed to be distributed? It's supposed to be the people own thing But the reality is that the internet is largely Power accumulated. You know, you know, I'm sorry. I think I lost you there. Can you explain why that wouldn't work again?

30:01 So you're saying okay, so let's see. I have three services. I have Google login and I have cloud flare Auto rag, which I assume is something to do with AI You would program in your little little you know your little cluster of machines that any request going out to Google automatically Susplodes it's not a cluster of machines. It's called donkey Kong or something I don't remember what he told it was like don't show on the architecture that donkey Kong shlong. Yeah When that thing's on it takes the cluster machines and says okay Yeah, there you go. Traffic that matches this redgex, which of course don't do that But let's just pretend you do it this way. Match this redgex. We just return an auto 500

30:36 Okay, that works on the Google sign and we fix the problem But the cloud flare request to auto rag you don't solve because that yeah request to Google happens outside of your Purview to like affect You need to have this big problem is is that you basically need a black hat team you need a black hat team to go actually crash Google and see what happens for the day and that's that's the only way to actually implement donkey Kong shlong It's crazy Almost been here for 20 minutes and Casey saying shlong and dong true I mean, wouldn't couldn't you also catch this too if like so I'm saying like basically any external dependency you guard with a red button That you can test right because then like then you could test

31:14 It's not so much like that you're depending on that to tell you when Google or or cloud flare or whatever goes down It's more like you're able to test like hey from a risk calculus perspective if cloud fl- Oh, but you're saying you don't know the dependency. I guess it's the problem It's the same problem. I was talking about originally just like if you don't if this isn't or worse yet I guess we're if you combine the two problems that we've talked about it gets even worse because like I'm saying like Oh, it would be nice if they said what they depend on But what you were saying the level is that like it's unclear that anyone at the company may even know the answer to that question always like

31:47 Dude forgot that oh, we actually do call like the Google service in this one place and we forgot about that And we didn't put on the diagram and so then we went down because you know blah blah blah. So yeah, I guess it's like what I'm saying is just an observation of like this increases This decreases your ability to analyze your risk and there may be nothing you can do about it Yeah, I mean, I think I'm in that team. I think I'm literally in the team of like that's just how it is Yeah, like either roll your own everything or forget it like I mean do companies do that to people just like I just don't want any dependencies And we just said that trash. I'm so happy you said that because that's the next thing

32:19 I wanted to talk about which was my happiest part about this whole thing is knowing the level of tweets that we're going to happen from DHH Right after this thing started going down So stoked I'm like here comes DHH. He's going to be coming in hard. He's going to be like on-prem solutions Exactly at what point do we just have it all under their own He was living the dream life. They're like, oh, it went down. Oh, I didn't know that He's tweeting a lot. I didn't see it going down. Was he going nuts? Oh, yeah, he said though There's even I saw more tweets about people getting worried about DHH tweeting than DHH tweeting But there were several And this is like my favorite that's my favorite aspect of the whole thing is that

33:03 Someone who has been right this whole time who everyone makes fun of and then the thing happens that he talks about and he's just They're all like oh great here. He's going to be insufferable now It's just like or he's just going to be right and he told you so Okay, I thought it's fantastic. We don't need to discuss that part I was living real life touching grass. I missed it all all right So I wanted to so something funny happened to me is that I was streaming during that day and I got done my stream I ended up like a I think it was the hour that cloud flare went down and that I had a whole bunch of programming that I had to do And I just proceeded the program until like three or four in the afternoon open up twitter and everyone's just like

33:49 I can't work the life's ruining the internet's down. I had no idea the internet even went down I was just in my own little world with my headphones on and I missed the entire like experience of the internet exploding Yeah, I don't actually know like what exactly went down like in my own Like what do I use every day that would everything? Everything with like I remember like people couldn't we like on our discord people couldn't upload images Could Tesla start could you drive a Tesla? I don't have a Tesla. I Used a computer that is connected to an internet. I mean I took a car that is connected to the internet I think that's like Casey you're on a computer that's connected to the internet buddy Casey. I used to have that mentality

34:28 I used to like refuse to do smart thermostat smart cars and then I just gave up I was like Don't do it. They're gonna kill me. They're gonna kill me. You know what I mean? Like I know what happened Your wife bought that Wi-Fi lamp. You uploaded a picture of it and after that you're just like GG my wife but lover did that she's actually the other guy which might walk in here while I'm making fun of her um She bought a lamp that was like IOT and to turn on the lamp you had to download an app that connected to the lamp over blue tooth Not hold on and then you had to put your Wi-Fi password into the lamp for it to connect to the internet to upload its firmware And then you could turn the lamp on exactly

35:06 Yeah, and then it break itself somehow and it's useless It's crazy man. So ever since then oh and also I had to punch a hole through my firewall for it to work I had to make a separate VLAN on my network and on that network I had to basically allow a block like traffic by country I had to allow a different country into my network for that lamp to work which is Pretty was it Russia?

35:31 China I love China I never heard of them. Yeah, you allowed China in for a Wi-Fi lamp that uploads passwords that definitely wasn't Worth it Need the lamp baby. Need the lamp. Did it at least change colors? It was a like oh yeah, it was cool. It was perfect It also had a timer in it that when you like oh it's 630 sun rises to 645 It'll slowly bring the room up to light and it'll be like and it makes like bird noises It's like kind of a rock. I actually have a clock. That's like does that. Yeah, but it broke so it's yeah Whatever I love how like no one can program without the internet like nothing you just said requires the internet at all But they're like you know what we need this lamp to connect to China

36:11 That's the only way we can sunrise in America is a server in China We need a convenient way to ask you for your Wi-Fi playing text password, but like you know Oh lift up you what it is. Yeah, man. It's crazy. Like GPS exists GPS has a time source just you use that. I don't know why I need the Wi-Fi access Yeah, honestly peak tech was the clap-on clap-off lights dude you walk it back. Yeah, remember those commercials?

36:36 Everybody old people know they're all gonna have been to like Like the sleep that was that was peak technology Yeah, dude, so I will say the fact that we have to sign in to something to use it If for me personally is single-handedly the most frustrating aspect of modern life Like anything that I purchased that requires that I want to throw it out the window and I recently got an automated dog food feeder for my dog And get this I want you to just sit down and listen listen to this There's no internet to it. There's no Bluetooth to it It has a record button so you can just be like hey dog. It's time to eat. Yeah It records your voice up to 10 seconds and you can have up to four feeding sessions a day

37:22 That's pretty cool. We have this automated. We have is we have basic technology. Yes exists We have this as well. Uh, we have this for our cats. It's we we were it says bon appatee tiny puss every time Like it puts the little let's go What Is it what like in your voice? There's a little like a We recorded we recorded that And then all the stuff the food He's Mario all of a sudden The best part about this is you have to remember this is an automated feeder And there's two of them because we have two cats and there's one for each cat and they These feeders go off You know four times a day or something. So if there's guests over right?

38:18 It's just like oh, hey, how's it going to look like in the background? I'm just like uh, it's kept You don't worry about it. Yeah, that's awesome. It's really cool. It's totally normal, bro It's just a camera. Okay, I'm weird about it. But okay, uh So this cloud thing we got to get back I've never heard anyone call a cat a tiny puss before Oh I I abused that so much all the time What is happening right? Yeah, I mean in exactly the way you're doing now like I just I love double entendre What can I say I just any it's always good for me like I always like when you can accidentally Accidentally make it sound dirty. I'm just gonna. I'm gonna like it. I'm gonna like that I swear TJ leaves for one stream. Just oh god. You're right. This is he would love this

39:20 Yeah, I don't know if he would love it, but He'd buy shut it down real quick. He'd buy it. We would have been a different topic I'm from New Jersey. I can't I can't help myself. I'm sorry. Okay, listen all right change it. Well anyway What are you gonna say about some stupid cloud? Um, honestly, I have no idea Just I guess okay, so low level learning We had a thing where you weren't allowed to plug in devices on the Netflix network. Yeah Why are you giving up? Why why not just not have like do you really need sunset lamp or can you just Turn on the lamp so my thing is like Oh, I say this For a long time lived a very paranoid life where I wouldn't do certain things And I wouldn't use modern comforts out of fear of being targeted

40:13 But then I learned that you wouldn't use like something to help us understand that no literally like a smart thermostat I refused like I and like me 10 years ago with not a driven a Tesla because like oh It's you know, lithium ion battery is attached to a computer on the internet and they could blow it up right like that That crap used to be what I would say, but then I realized if you're being targeted They're gonna get you dude. It doesn't matter. That's really my way of life. Yeah, what what That's the one password I have faith in humanity Until it turns on me. So like if I go to like a public space I'll just literally leave my wallet just standing right sitting right there. That's crazy my kids. Hold on stop

40:52 Wait, okay, so I really just put my walls like right there and open like I'm just like no one's gonna steal it like I have faith in you man So far hasn't yet. Oh my goodness. Just jinxed myself But like I leave my stuff like this is like at like gyms that I go to like where I kind of know everybody But like I have faith in humanity to where like if they're gonna get me they're gonna get me if they want to get my wallet They're just breaking to my locker Because if they're gonna get me first off there is a shadowy cabal that's gonna get somebody for a very specific reason Leaving your wallet out for anybody to steal is not smart Okay, okay, here's a better example. Sometimes I don't lock the the door to my house like if they're gonna get if they're gonna get me

41:36 They're gonna just break the window kick the door down like You look at California though. That's different. Yeah, that's what you guys do out there You know, just read your keys on the car Oh, saying is it they're gonna get me they're gonna get me I'm gonna live my life. No, but that's that's right It's like it back to your question prime like a corporate network has more to lose, right?

41:56 So like maybe there should be rules in place to not plug in random crap But like at my house like I'm gonna do my best I'm not gonna go on freaking teamoo and buy a router from teamoo for $14 and then like be surprised when I get hacked I'm not gonna do that, but I'm also like I'm gonna use the internet lamp You know what I mean like I'm just kind of just whatever. I'm over the paranoia for just for myself personally Have good hygiene like use like shower. No, but like use use a password manager Right do all that stuff like VPNs went appropriate, but I mean like I'm not gonna like not use certain technologies out of fear anymore Just just how I've decided to do an ass like like like to have to sign in

42:31 Via Bluetooth send a password so it can sign into your network just to turn on the lamp like that's not even about getting hacked That's just an ass pain. I don't want this more like ever. Yeah, I don't care how great the sunrise and bird calls are and how So did you hear that that's actually a Southwest To a region like I don't even care like there's no money. You can pay me to sign into a lamp to make it turn on That wasn't like the far end of the spectrum. I agree, but like you know Counterpoint that is the legitimate counterpoint because it's like what you're saying makes total sense Like if you're if you're threat model for why you stuff in your house isn't gonna work is that like you know

43:09 Some some government security agency is gonna target you It's like forget it because they'll just come to your house with guns and take you right they don't need they don't need to hack your lamp It's not necessary, right But the they in your sentence could just as easily just be incompetent people at the actual manufacturer Who cannot get the freaking lamp working and that's why I don't buy these things like I know that if I bought a nest thermostat I guarantee you the temperature in my house would you random all the time and that is because I also seem to have uniquely bad luck with this sort of thing So you know the all of people were like what are you talking about my Linux install work the first time right?

43:46 It's like for me. It's always like I always will hit the weird case for some reason So if you know if I bought a Tesla it would the software would never be working It would be and in fact a friend of mine actually had this he had a Tesla pretty early and he had so many hilarious stories He's like yeah the air conditioning to stop working for several weeks because of some software update But then it started working again now and he's like oh the sunroof stop working because they updated the software and they changed I guess new models have that plugged into a different Heart physical port and they stopped sending the things out the old port So now his sunroof is just closed permanently like us and I'm like yes

44:19 That is what would have happened to me if I about a Tesla. I don't want any that I just want to know that like the sunroof is on a motor and unless the motor dies it will open right? Yeah, it's less failure points I will say like I don't like being on the cutting edge of new technology So like I own a Tesla for example right um into that point I recently went to go drive somewhere I sat in my car and then on the Tesla It said there has been an electrical failure call Tesla and I'm like what exactly is that and so what I had to do is have them To the Tesla 45 minutes away and fork out $5,000 and because it's Tesla it's proprietary So they can't tell me what happened. They're just like yep. Sorry. We fixed it. Here you go. Give me your money

44:55 And so I feel you in terms of the the ease of use on technology Things that are well solved problems. I will use newer technology, but like electric cars I have a Tesla. I'm probably gonna gonna we're gonna get rid of it for that reason I don't like effectively being the the market test monkey right for these bigger companies And they're not serious about fault tolerance like that's the thing It's like this car was not I mean maybe the brakes were which is good But like the actual ability to drive the car was not as far as I can tell right and use the features of the car Yeah, I would imagine like internally there's like some real-time hard System for breaking and stuff, but like the ux of turning on the car

45:35 Not so much. Maybe not. Yeah Sorry, you go trash. No, let's go say you just go. Okay Prime is so mad. What does that happen? I don't know. He's melting down. It's not mad. He's melting like so disappointed enough I just want to get this off my chest so disappointed at us right now all I want I don't really care about smart houses or anything all I want is The fridge that has the clear door so I can just look inside of it without opening the door Oh wait Is that what you know the purchase where you can see before you open the door? It's all I want They have that you can buy that. I know, but it's just glass Why don't you just get triple paying glass or something like what why do we have to have technology that involves a

46:18 Camera when we already have something you could see through called glad. Oh, no. That's what I'm saying. I just want a simple glass He wants the glass one. He's agreeing with you. I don't want the technology. This isn't even about technology I'm on your side for When I hear everyone like all the nest thermostat is good. I'm like bitch. It is I sent my AC to 69 and I never said it again That's right here What the hell do you need a smart Can I tell you can I tell you okay? Okay. Okay. Let's get us get us up This is the most American thing you're ever gonna hear, but here. I'm laying in bed. It's nine Super my son two years old as a sleep I can see on the baby monitor that his room is a piping hot 74. Okay. I don't want to get up

47:06 I'm in bed my brush my teeth my viselines and you know what I do? I go to my phone I can change the temperature in the house from my bed. I don't have to get up, baby You can stuff like you're ready to change the temperature I turned off the camera and only listened to audio And for most of our kids childhood, I'd even have one of those I slept near enough that I could hear them cry if they got really honoring and that's it because you don't need to listen to everything Probably good practice Just stop it you don't get 74. I guess sucks, but that's totally normal temperature The story is I changed that ship for my phone, dude. It was great. I don't have to get up. So that's why Crying just more he didn't know yeah, I saw in chat and then good

47:57 I went down and my baby froze to death But you know that one day that I could change the temperature from bed. It was all worth it, right? Like Google law didn't stop working. Yeah, I couldn't he I couldn't show the house is 89 today in the summer Primer you mad because you bought a Tesla Me no, I never bought a Tesla. I have a vegan said you bought a Tesla. That's why okay 2006 What's called chevy flat bed pickup that has manual window roll?

48:23 Dude I said when I'm in up with prime and t's in LA they saw my car and prime was so disappointed And he was like and he was like he was like you need to get your priorities straight in life Oh, what was the car? I have a Porsche and Did he just he just looked at the ground was like dammit trash Now Let the man have my bad dude. That is a crazy message holy shit We lost the baby because I had an upstream depth on gcp Dude, but that's like actually okay That is like the world we live in now like we are live in a world where computers have crashed planes into the ground and everyone died That's not like that true that's terrifying So like like it really the people underestimate the risks from actually

49:19 Increasingly putting technology in places where they could have bad results I agree the baby froze to death is probably an exaggeration in this case But like things like that are not far right and it doesn't have to be about security exploits It can just be incompetence. That's great. You guys have heard of the the Toyota 2009 case, right?

49:39 Yes, I'm deeply familiar to I was like what 16 year I don't know So I don't know if it was just Prius Casey and that's because a lot of cameras too But basically like so the way the ECU worked on Toyota is it had an arthas in it and inside of arthas As you have like the thread control block for all the threads that run on the arthas Timon Timon arthas real-time operating system real-time operating system. Sorry Basically the way that it was designed if like a single bit got flipped in the tcb The task that ran the brakes would not pro or no, sorry that ran the accelerator would not properly halt So like the cars were literally just randomly flooring it And there was no way to stop it because the way that they wrote the the braking logic because it was a fly-by-wire braking

50:22 It wouldn't actually actuate the brakes So people from like 09 to 2012 I think were like just randomly getting caught doing like 120 miles an hour and find into a brick wall because the ECU was like crashing non-gratically I didn't know that they ever root caused that that's always they actually found they actually did oh We shouldn't say because it I didn't know they ever published what it actually was so it was it was closed privately But Michael bar who was the consultant that was doing the analysis has published some of his slides of his findings I think a lot of it is still under NDA with Toyota But he basically like published what he thought it was before it went public public internal right

51:01 And he said basically yeah, like this thing is a land of mind if a bit flips here here here here here The ECU will just run on forever in undefined behavior and that includes like oops the brakes don't work Wow That just recently like just this year maybe last year Tesla had it where there's some sort of fire that happened And people couldn't unlock the door because there's a power power failure in the Tesla And it just like the door shut and the people burned to death inside their own car Fail open is very important In fact in fact We can tie this back here at the because we are at our hour mark I think or getting close to it We can tie this back to the original topic of the day one of the things that Google included in their like

51:41 Mayaculpo we took down the entire internet. Sorry about that are bad Was at the end they were like what are the you know they like what are the steps we're going to take to try and improve our process You know and one of the things they listed was this service should have failed open like instead of the Default for when the quota system goes down being deny everything Maybe it's just approve everything so the services can keep running I don't know how that necessarily but maybe you could do something more cabin to where it's like Approved any reason a bit like approve anything that doesn't allocate news too much new space I don't know what they meant by that exactly But they were like try to make these things fail open so that it won't just take down the internet if like the quota check fails

52:21 That was what I assumed that they meant but I don't really know because again it was very terse statement and I only read it for this podcast right I wasn't I wasn't like I don't study this stuff so I'm not sure but so fail open is a pretty important principle like what should something fail closed Or something should something fail open and if you have like door locks on a car Ideally if there could be people inside you always want it to fail open so that there's no way for like a car Computer system to fail and then trap the people on the inside you want to fail if you know if the system for locking goes down You want to be open right because worst case people steal your car or something like that. That's a lot better than burning alive

52:57 It is tough that Okay, so it is tough from like a I think a design perspective It is because the control plane software that crashed was the authorization software It's like I agree with you it should fail open but like to your point what does what does fail open mean in an authorization case Right you can't be like yeah go do it whatever because that's bad But inversely denying everything killed the internet for a minute. So what do you do? It's tough prime guy I was about to say you know this fail open thing that also like there's there You also have a problem with this even with just the physical reality remember how Tesla was claiming that they're Whether or not it actually is bulletproof their windows were bulletproof

53:32 What happens when you drive a cyber truck into a pond and you need to break out of the window Actually have a keychain right. Yeah, I have a key. I have a key utility that breaks windows But normal windows they just you hit it once and they explode into a thousand little pieces What do you do when you hit it and it just goes Yeah Yeah, well, this is like the locks right it's like one of the reasons that you want a lock to always be possible to do Physically if you could right is for this reason It's because if you think about what fail open means Well, if a mechanical element has to push the lock up and down. It's not accessible to humans It's almost impossible to build a fail open version of that because if the electrical system fails the locks cannot actuate

54:15 So there's really no fail open possibility So like again, like just smart mechanical design is like your car should have a way that you with your hand Can unlock the door from the inside if you needed to right and like anything short of that is just kind of bad safety design Right and the same is true for internet services to the extent that you can do them You want to be thinking about that the same way it doesn't quite analogize exactly, but it's similar right That is interesting because like the Tesla you know you have the way that the doors work is you have a button that you push that Electrically pulls a window down and opens the door for you, but there is a physical I'm pretty sure it's pure mechanical knob where you can rip that and it pops the door open

54:52 It's bad for the windows obviously, but that's that's the fail open scenario. It was like, you know You are in a lake the electronics are dead pull this lever the door will open So I'm interesting interested in that case if they like if they knew that if it was a rental Maybe that version didn't have that model. Maybe it's not mechanical. Yeah, like how did they burn how did they burn to death without being able to do that?

55:10 Yeah, I don't know that's just the The things that I heard I don't know how to exactly works. I didn't do any research Window wise is worse because like to your point like you can't open the door if you fall into a lake because the water pressure is too high So you need to be able to open the window and in that case, I don't know what their answers to that break it as yes or I don't know That's a terrifying scenario. It is terrifying. No, I know you love rust Yeah, yeah, you think that the GCP outage would have happened had this binary written rust Uh, you know, that's actually a really good question I think that we would have still had some sort of weird state because at the end of the day How do you have a state? Let's just say the YAML file produces something incorrectly

55:50 What what happens to a GCP service if they update the policy and there's nothing there in a thing that's supposed to be there Right like someone would have probably programmed the positive case and then what happens at the negative case It just wouldn't deploy like all things would stop anyways. I'm not really sure like how this could have saved it much other than You would have had a pre-planned. Hey, it's gonna explode Maybe you could have identified it earlier and maybe the outage would have been Less I mean, I'm all for I'm all for the idea of unique pointer slash smart like smart pointers where you can't access things without Some sort of like hey, yeah, this thing's not new like I like I'm moan ads. Yeah, dude. Give me in

56:28 Let me let me have a piece that moan ad like that. I'm all about that But at the same time. I don't know because the real question I guess the question I'd have to follow up with is that it's not whether or not rust Would have made this possible or not possible? It is if it were written in rust Would this software ever have existed beyond somebody's idea or not?

56:49 And that's really the question I have to ask is can it actually exist in production? I'm not too sure right We don't have any security exploits in rust because the software did not get written right because it never got shipped It's never in production Yeah, that's thing right like pee. I so whenever I do videos I just say at the end of the video about like memory exploits like would rust have saved this I think the answer to this one is Sort of right like because it wasn't like a necessarily like a memory corruption vulnerability or memory corruption issue It was more just like you dereferenced a bad address I think the rust programming model would have helped with it But like you can do an unwrap on an option that's none right and then like you crash

57:26 So it's like there are still human factors and the code that could have caused it to be the same DOS condition if you know the the ha ha object in the yaml file wasn't there right So it would have made it I think less what was that they would have had an expect right like because they like they'd been like that You cannot have a license or whatever it was without these fields and they probably would just expected Sure, which would have caused lit it's like no different than a nail pointer crash because you wouldn't have known about it until runtime So I assume it's the same thing. I don't think rust would have helped it all here Yeah, it would have been nice if they had said What language it was written in shows a little bit of the code like

58:03 Told us if this was Gemini right this could have been like somebody vibe coded this and like hey man I'm pretty sure the AI said it was great You know like but there was no information so we don't really know like human factors wise We don't we have no idea what happened to really because they were very very Like it was actually written in rust as my guess and they actually don't want to release that information that it was written in rust By Gemini there's written in rust by Gemini Yeah, it's part of the greater agenda of the rust industrial complex. You sheep will need to wake up Put up the more where's the board with all the connections between the rust foundation and the like where is it your diagram?

58:40 We need the little conspiracy diagram All right, oh this is are we done are we have we wrapped I think we're wrapped right we got tons of great content in here Do you know what could be unhappy with that no one could be unhappy with that episode Child you just you that there was there was a golden episode. What do one more do you want from us?

59:00 Little level little level everybody Errors on my screen Terminal card