The Secret of the AWS Outage

0:00 Does everybody ready for this bad boy? I'm sorry. I don't know this topic we picked. Dude, it's gonna be so exciting. You don't even know trash, because you obviously didn't read the email that Casey sent. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah. Anyways, I'm sorry.

0:15 All right, today on the standup, let's see on. Casey, we're doing your topic. I'm very excited about it. I just cannot wait. All right, today on the standup, we are gonna be rehashing an older topic, the AWS or the Amazon out of shit, just recently happened. It was just a couple of weeks ago, I forgot the exact date and forgot to look it up.

0:32 But Casey apparently got the inside scoop and actual Amazon engineer came up and said, hold your horses, you need to sit down. No, no, no, you're jumping ahead. That is not what happened. It's, it's easier than that. It's easier than that. Okay, why is it easier than that? All right, hey, Casey, why don't you do the intro?

0:50 Today on the standup, Casey will do the intro. Oh, I started fresh, I love this. Yes. Today on the standup, Casey is going to create an urban legend about the Amazon outage because of a ridiculous series of events which I am going to recount to you now. In the longest, most boring possible way because this is an entertainment program where we stretch everything out.

1:14 So here's the deal, if you are familiar at all with sort of the tech scene here in Washington state, it's all kind of around the Redmond, Washington area where Microsoft originally kind of, well, they didn't originally settle, they moved up there, right? And so there's a bunch of tech around here now and there's a couple of places.

1:36 It's called the East Side because it's not in Seattle proper. In Seattle proper, there actually are Amazon towers now. There's a Google campus there. There's a Facebook campus there, et cetera, et cetera. So there is stuff in Seattle proper. There's also a lot of tech on the East Side, Kirkland, Redmond, Bellevue, et cetera.

1:55 I love Kirkland. That is the scene. A lot of groceries from Kirkland, big fan. Kirkland, Kirkland signature. That's named after that Kirkland. That is the Kirkland, because Costco's from here, yeah. Prime, let's take a field trip there. I know, I would actually do it. I'm totally down. I just got a Costco membership.

2:11 Kirkland is out there. We would love to be sponsored by you. I would just throw it out there. There was one brand I would hitch my wagon to, it's Kirkland. I don't have my wallet, Kirkland, I don't have my wallet, but I've been an executive member since 2003. Yeah, show both sides of the card, Prime.

2:28 2003. I get Kirkland stuff at Costco. This story will quickly move its location to very close to the Costco headquarters. So that's how sprawling and epic this tale is that I'm about to tell. Oh, good, I'm invested. I'm ready. All right. Jeff Roberts is going to think I tell the story horribly. He's a great storyteller and tells stories in a really great way and I suck at it, so he's going to criticize this.

2:54 I'm sure if he ever listens to it, hopefully he won't. But so here's what happened. I'm at home and evidently, I was playing on my piano keyboard too hard and I broke the little mechanism inside the C4 key. So now when you push it, it kind of goes like clank, clank, it makes a loud clicking noise. That's an important one too.

3:23 You hit that key a lot. See a fellow I know calls see the people's key. It's all the people's scale, right? It's a people's scale. So C4 key, you need that key. It's right in the middle. Anyway, so I'm like, well, it occurs to me that I probably will want to get this replaced at some point. And because I have to do music work on this particular project, it's not great if I don't have a keyboard to use for various things.

3:51 So I'm like, I should have a backup keyboard. You know, I made a lot of mistakes in my life. I had billions of dollars of venture capital. Legions of engineers under my command, tokens on tokens on tokens. And what did I decide to do with it? Did I fix reconnecting web sockets? No, I shipped an electron app.

4:21 I failed you. I didn't mean for Dan this way. Not even the walls of this prison will hold us back. The love of a founder! You can't stop us. My biggest regret of all is that I'll never program with you again. Tupel, we still can. Tupel, a native app that lets you code together, even from CodeJail. Free yourself by going to tupel.app and use CodePrime for 90% off your team's first three months.

5:22 Casey, can I ask, this is not what took down AWS though, right? Just to be clear, who knows at this point? This is what mashing of the keys was so loud. I didn't know if it was a load of key space around the world. I'm trying to fill an hour with a story that's got five minutes of content, all right?

5:40 Just bear with me here. Anyway, so what happens is I'm like, okay, we could get a, or place the keyboard, the guitar center in Redmond has this kind of keyboard. I will go there and get it. That's it. That's what was supposed to happen. That's all that was supposed to happen in this evening. I'm following.

6:08 I'm following. Yeah, okay. Got it. Get in the car. Go to Redmond. Actually, at dinner, go to the guitar center and, sorry, we only have an hour TJ. We can't go for all. Okay, bye. I ate dinner at a place called Musashi, which is this like sushi place that's right in Reb and Ten Center for those who know the sushi.

6:35 I want that. Nice. Anyway, I had a sake down. It's like I had a bowl of raw salmon. There you go. Best fish by the way. Best fish. That's right. That's the best fish. Yeah, I was just like, thank you. Best fish. Everyone knows, everyone knows this is the best fish. Casey loves it. I love it. Orcos love it.

6:51 It's the best fish. Orcos. A million orcas can't be wrong. It's probably only 60 left. But whatever. 60 orcas can't be wrong. So anyway, I go to the guitar center, I'm like, I, I placed this order for keyboard and line. They're like, oh, yeah, like our system doesn't really work at all. So we're going to need a while to like figure out how to process it.

7:11 I'm like, okay. So I walk into the back of the place where the keyboards are and I'm just like looking around. So this is when a guy comes up to me. It's actually two people. Like a guy. I think it was, I think they were like, you know, maybe not husband and wife or partners. I don't know. They were together, right?

7:30 So this guy and his girl come up. And the guy goes, are you case of your Atari or is that something like your case of Atari right? And I'm like, yes. I am. He's like, I saw you recently talking about the AWS outage on YouTube and I'm like, oh, let's go. Yeah. And I'm like, he has got motion. That's right.

7:50 Am I famous? Am I famous now too? Yes. You're famous. And we got it. He lives is famous. Let's go. It's everywhere, bro. The whole world is watching your snacks trash. The whole world. Please. Not for me. Sorry. I'm like, I'm like, oh, do you watch the stand up? I'm like that. Yeah. Like it's really popular.

8:13 I love that show. I'm like, thanks for watching. And I thought that it was just somebody was just like, I saw you stand up. It was cool. Thanks. And I was going to say thanks. And he's like, but no, he goes on that program, you said if anyone had any more information about what happened in the AWS outage to tell you is like, I have something to tell you.

8:37 I'm like, I'm like, oh, really? Do you? What did he open up like a secret door? Yeah. Kind of. Kind of. This story just gets so hilarious. So anyway, so if you remember this outage, it was a dynamo DB outage issue. And eventually when I think when we covered it, I don't think they had actually published their postmortem yet.

9:01 So we were just like, we said, like, we don't know, like, it's kind of weird. Like, what happened? It was unclear. There were some vague things about it, but didn't right. They published a postmortem later. And the postmortem was really kind of confusing because what it claimed, and I'll recount this for people now, because we never covered this on the program.

9:19 So if you haven't read the postmortem, you, you wouldn't know, even if you watched our old show, the postmortem said that what happened, the root cause, not the like fall over effects of dynamo DB going down, which was the toll other thing apparently, that the root cause was there's this thing that's like a DNS planner, they called it, a thing that makes a plan that's what like hosts should be contacted for what regarding dynamo DB, I guess, right?

9:48 And these plans are updated fairly frequently, they didn't really say how often, but this machine is like cranking along and it's like coming up with these new plans. And then that machine apparently doesn't do anything. It just comes up with the plan, like that's it. And there's these other machines called DNS inactors, and I think they're machines, I don't know if there's machines or processes, but there's these other, you know, independently writing things called DNS inactors, of which there are multiple that take these plans and go and set all the DNSs or something.

10:20 They were not that clear about it, but that's what they described. Is this sound right to everyone to do, I don't know if you guys ever read this? I didn't know the postmortem was like a race condition thing, right? Like, when it just took too long. Yeah. We'll get to that in a second. This is the structure of it, right?

10:33 Okay. And what they said happened was normally a DNS inactors, right? It goes through and it says like, all right, if I'm going to, like, I was told or I grabbed plan, you know, 12, and I'm going to go around setting plan 12, I'm going to quickly check to make sure that we aren't on plan 13 yet, right, or 14 yet.

10:55 So like, I'm just going to check, quick at the beginning, what's at the beginning, it's said, right? I'm going to check once at the beginning and I'm just going to go and then if, if I've got the newest plan, I'm just going to go through setting all the plans and then I'm going to finish, right? And what they claimed it happened was one of these DNS inactors got kind of hung up somewhere and another DNS inactor that started up, started updating everybody to a newer plan, right?

11:25 The old DNS inactor, though, started again and put in kind of the old plan back in. And then they hand-wavingly said something like, and then start because it was trying to clean up the plan, it like deleted something, like deleted the configuration, leaving no way to recover it. And like, when I read this, I was like, that doesn't make any logical set.

11:48 Like, I don't, I don't understand what you're actually saying. So it was like, it was an unsatisfying explanation because I'm like, I could imagine something like this, but it didn't all click for me and I'm like, I would like something more detailed. So does that all make, you know, you see where I'm confused about the official explanation, right?

12:06 Yes. Yeah. All right. So that's all I knew. So this guy says, well, look, this is what happened or rather, this is what happened to me. We, meaning him and the woman that he was with there, it's like, we were at this place called the black duck cask and bottle. Now this is a place, this is a, like basically like a gastropub, there's what they would call it nowadays, what the, what the kids today would call this thing.

12:36 It's like a place where you go to drink, but they also serve like what is supposed to be upscale food, right? And this place is in downtown Issaquah. It is literally like just over, like there's a highway, highway 90 runs in between literally where this place is just in the other highway and the headquarters of Costco is right on the other side, guys.

13:02 Like the giant new headquarters that they built is right there Kirkland signature, the plan for what Kirkland signature items will be coming out next year is happening in that headquarters. If you have a guy's over one to come out and visit Washington, we can take it. We can knock on the door and say, Hey, we're podcasters.

13:24 Can we come in? Can we get pizza? Yeah. Can we get pizza? Pizza? Yeah, yeah. You buy a Costco pizza and you buy a Costco hot dog. No, no, no, no, no, no. I will not do the relish. Do not do relish on top. It is not, it is nothing like pineapple. It is very terrible. But pizza dogs are really like they're the words at what you did that you put pizza around a hot dog and then you put relish on it because you were just like, yo, that's why I'm doing this.

13:53 Yeah, yeah. That was terrible. But like pizza dogs are really good, though. So it's like pizza to blanket, but like a pizza. Yeah. Exactly. Pigs in a pizza. Hell yeah. I'm going to try that. Yeah. That's right. I was like, what? Okay. I'm like, I'm not sure what really happened. No one knows because I'm, this is where we get the information.

14:12 So this guy says, I was at the black duck, duck, cask and bottle. And we were sitting next to a table where two guys were talking. And one of the guys in the middle of the conversation goes, so you know that Amazon outage that happened yesterday, I apparently that was my fault. That's how he starts the conversation.

14:37 Is this like a spy movie? Yeah. This is like third. This is why I say we have no idea if this is true because not only am I hearing it from someone that person wasn't even supposed to know this. They just overheard someone else talking and maybe, maybe the person they overheard was like a pathological liar who made it all up.

14:55 Who knows? Right. But here's what happened. Oh, right now though, that guy who is at scary name gastropub. I can't remember what it was, but something about death. He's sweating bullets right now because he's like, no, he's not, because he doesn't work there. No, no, no. He's discets better. This gets so much.

15:15 Just wait. Just wait. I promise you. Let's go. The story, the story is so good that I'm telling it even though we have no idea if it's true and it probably isn't, right, because it's such a great story. Yeah, I'm ready. Anyway. So this guy and this girl are sitting at the table enjoying their meal at the black duck cascan bottle.

15:39 He, by the way, the guy at least, I don't know about the girl, but the guy at least works on cloud stuff. So when he shares this, he's immediately like, oh, right, so like, and she also apparently was very interested. I think they might both work on this kind of thing. I don't know. Because they both are like, wait, what is going on?

15:58 So they completely stop talking and kind of like, you know, are trying, it's very loud in here, but they're like, let's listen as closely as we can. So the dude's like, yeah, apparently that was my fault. And his, the person across from him who he's eating with is like, how can that be your fault? You haven't worked there in like 10 years, right?

16:21 And the guy's like, yeah, exactly. But apparently there was this code that I'd written back in 2012, the end of the, the end of the world back in 20 calendar was about this is it. The whole thing didn't go off as planned, but back in 2012, I wrote this piece of code that was just supposed to be an easy way to copy the configuration file from one machine to another because I was trying to be able to like set up servers very quickly and not have to actually manually configure them myself.

17:00 So I wrote this thing that was only for me to use. And it was supposed to just do this simple copy operation. It's not a production thing. It was never released into anything. And it was just some, some code that I had used. And he's like, well, yesterday, or you know, whatever it was yesterday, I got a panicked call from Amazon.

17:26 And they were like, okay, this code that you wrote is about to take down the entire data center. And we need it to be stopped like immediately. And he's like, what are you talking about? I don't work there anymore. I haven't worked there in an incredibly long time. And none of my code would be running in that thing.

17:46 And he's like the person on the other end of the phone is apparently like, no, like this code, it's got your, like we looked at like the history of it. It is your code from like your repo. And he's like, yes, okay, I did write that code. And he's like, well, we need to be able to stop it because it's deleting all of these configuration files.

18:05 And we don't know how to get it to stop apparently. This is literally like what the guy is saying. Again, I have no idea how plausible this could possibly be, but the same idea, I don't really know why these data centers go down for seemingly weird reasons that don't make a lot of sense in the post-mortem either.

18:19 So I don't know who the problem is. I don't know who the problem is. By the way, this, this sounds like standard operating. Okay. Like Fortune 500 company, right? I'm a lot of this person. So the guy says, apparently they found that configuration copying code in the repo that I wrote did maybe some light testing on it or something like this.

18:37 And then must have started using it to actually copy the configuration files from like different machines to other machines. But I guess it had like a bug or something where sometimes it would just start deleting the configurations after it did the copy. So when they started actually deploying it to things, it would start like it would do copies then delete things and leave them in like dead states that couldn't really be recovered.

19:00 And so they're like, we need the password to something and apparently like they couldn't tell from the conversation what the password was actually to. But the Amazon people are like, we need the password to this thing in order to like actually go like modify whatever it was, the build, the bucket, the permanent storage of the thing, the root password for some container they were running.

19:22 I have no idea, right? And so he's like, okay, I do remember that password. And this is the part where it ties back very tightly to the standup again. I do that, I do know that password because I use the same password thing for basically everything. No! It was me. It was me. Oh, baby. It's slightly better than Primes, which is that he has a different password for every month and then he depends the year.

19:51 That's a dumb strategy. I'll be better strategy than that. So the password, because he remembered that the code was from the fall, the password was wishbone 12, because it was 2012 and wishbone for Thanksgiving. So that passed the password, I guess, to this thing. Okay, so he remembers the password is wishbone 12, but he doesn't want to tell them because he's like, how do I know who you guys are?

20:20 You're just calling me. I don't work there anymore. How do I know you're even from Amazon? Maybe this is a fishing attempt like what's going on? And why don't you guys already know this thing anyway? What do you need this password for? Like, I don't believe you. So he doesn't tell him, it's wishbone 12, he's like, okay, here's what you should do.

20:41 If you can tell me the interviewers that interviewed me for the job at Amazon when I went there, when I was actually working, when I was contacted for them, the people who were on the phone screen or whatever the hell it was, if you can tell me who those people work, because the HR department will have a record of that, then I'll believe you're from Amazon.

21:01 And the guy says that then he hears the Amazon go, like, get me HR on the phone right now. This is like the president's on the phone with my grandma out of retirement. Please tell me their HR service was down because the day that would have been amazing. It would have been checkmate, that would have been amazing.

21:20 So is this why it was down for multiple hours? Is that they were just waiting for HR? No. I don't think so. I don't think so. I don't think so. I don't think so. So they did apparently get the names, they told the guy the names, he's like, okay, here's the password. Wishbone 12. Wishbone 12. Wishbone 12.

21:37 It couldn't have been so much worse trash, it could have been so much worse. Yeah. He likes the bathroom? Yeah. He likes the bathroom 12. Yeah. So anyway, so he gives them the password and they're like, okay, and that's the end of the phone call, right? So apparently, and now, like I said, I have no idea, I have no way to verify the veracity of this or not.

22:05 I can say that the official explanation didn't make sense because it didn't sound logical to me how this would have really taken machines out because couldn't you just run the planner in an actor again, it'd be really weird because they are reaching in an updating configuration. So it was just, it was like, what I will say is the idea that somebody who didn't know what they were doing pushed a copy thing to prod that then went rogue and started copying configurations to things completely erroneously and they didn't know how to stop it actually sound slightly more plausible to me, believe it or not, but I have no idea and I also don't know why you would need a password for that.

22:44 That part didn't make a whole lot of sense, but I could imagine if it was some kind of containerized thing, if the password would be more efficient to like edit something in the container, I have no idea. But why that would have been set by a guy a very long time ago, that doesn't make a lot of sense either.

22:57 So now I have two conflicting stories, both of which don't make perfect sense and I don't want to make of it. So the conclusion of this for me is, can I get another random person to tell me a corroborating story for one of these two stories so we can know what happened. Dude, I want to meet Wishbone.

23:22 Okay, that's this code name, I want to be in there, I'm sorry that we doxxed his password strategy because he probably is like, what America for July? That's America 12, our America 26 for this year. So yeah, oh crap, you know what, Wishbone 25 is the password to a bunch of things that Amazon, or not an Amazon for this guy's, this guy's poor account.

23:46 So I do apologize, I probably shouldn't have said what the password actually was. But don't worry, we won't fix it in post, it's too funny. It's out there now, I apologize Wishbone 12, but if you can anonymously confirm or deny this story, that would be very helpful because now we just don't know what to believe.

24:05 Honestly, from all my experience, this sounds very, like everything but the password part sounds extremely reasonable. Okay. That's the same with me, that same, same, I had the same opinion actually. You can't copy my opinion, I said it first, okay. I was going to say the same thing, I was on mute. I was kind of just illustrating this whole thing like in my head on how it was raining outside, storming, it's a crazy phone call.

24:33 I just love the fact that it all started because you broke your keyboard. Yeah, none of this would have happened, otherwise. We would never have known the secret story, the secret true story, the secret of the sign of Mojita out of it. I don't understand why they wouldn't publish that, honestly. Just be like, yeah, we had legacy code that was locked behind a password wall in which we had to figure it all out and restore from a very long time ago that was deleting files and production that should never have been there.

25:04 I can tell you why. Why? It sounds terrible because the MBAs who read that will think that that makes Amazon the worst software company in the world, even though it makes them above average. Oh, no, oh, no, oh, my gosh, well, that was a beautiful story. I'm actually, I'm very happy right now. This is not just made my day, this is, this is made several weeks of my life.

25:33 I'll be giggling to myself for the next while. I was like, I don't really care if the story is true or not because I loved the whole thing. The whole thing is cracking me up. And I also like, it did remind me that I don't like, I really wish even, let's suppose that the official explanation is the actual accurate one.

25:53 That is what happened. This story was just a random guy who was talking at a bar and was just making stuff up right that someone happened over here, totally plausible as well. But I don't understand the pathology, like the pathology they explained for the dynamo to be thing just sounded really weird because I was like, it didn't, like, it didn't add up to me.

26:15 And it, you know, it's entirely possible that it's like, well, it's because we've like abstracted away the details that would have all made sense where you're like, yeah, it's like, it's a lot hairier than what we said. But what we said is basically true, right? I'd still love to hear that too. So if any, if someone in Amazon ever wanted to come on and actually explain like, no, here's why the thing could not just run the DNS and actor again and everything was fine.

26:37 Like, you know, that would be cool too, honestly, because I'd love to hear that. That love that I'd love to hear the more technical explanation for what they act. I would really like to. Yeah. Because there was some guy made a testing utility for Google a while back for like launching things in specific regions and then do some sort of auto deleting and all this and then he accidentally deleted like Australia's pension.

26:59 If you remember that, this happened just a few months ago. It's just like, we, this just happens. Someone makes a tool that's like, it's great for testing. It's just for me for being able to launch and make my test work really, really fast and someone accidentally runs it and prod and forgets a hidden magic CLI flag.

27:15 And boom, 90 days later, the entire Australia's pension was deleted. Wow. I can find out exactly. Exactly. Yeah. It's basically like, wow, this thing makes it really easy to change environments. That's super helpful. That's like, oh, well, it makes it super easy to change environments. Okay, so how long was this whole encounter?

27:36 Oh, great question. Not that long. I mean, it was just, they literally just came up and it's like, I want to tell you this story because I like that they sit down with you and like, no, we were in a guitar center. Oh, yeah. Yeah. Yeah. Casey was playing stairway to heaven and he now the guy came up and just loved his tunes.

27:54 It was great. Oh, by the way, so Google it wiped out all the data and all the backups as well. What? $135 billion worth of pension accounts. Are you kidding me? Luckily, Australia had a secondary cloud service to store things in or they would have just been like S.O.L. Who was it? Someone in the australia who this was a Google person who did this?

28:18 Yeah, yeah. Google had a testing harness that would just like help launch up environments and all this and if you forgot to set a TTL, it set a hidden TTL of 90 days. Wow. If I remember the story precisely correct, yeah, you need super. Yeah. So just 90 days later, it's like, OK, I know where all the stuff is and where all the backups are.

28:39 I'll get them all out of here for you. Don't you worry. I can delete those. No problem. I do think like this is this is a scary thing that you learn when you first start working at like a company as opposed to just like, you know, programming, you know, on your own in school or at home or whatever you were doing is like, if you write something, there no matter what, no matter, it does not matter.

29:04 Like if you write something, there is a non-zero chance that it will be used for potentially decades. It doesn't matter. It does not matter if it is utter garbage. It does not matter if you tested it. It does not matter if you put a thing at the top that's like, do not use this. It will destroy the world.

29:19 It does not matter. There are people who will just deploy stuff. Yeah. That is an impossibility for me. My code is definitely still running in production and you're 10 years previous. I was at a meet and greet and someone came up and said, hey, I work at Netflix and I was like, oh, he's like, yeah, I've seen your code multiple times.

29:36 I'm like, years later, there it still is. All right. Let's go, baby. It does happen. Oh, by the way, I personally think this is why A.B. testing sucks. I'm going to throw this out there, hot take coming in is because with A.B. testing, you kind of go like a little bit more Rambo style, a little bit of cowboy coding and you go out there and make like a less supreme experience because you just want to be able to do this kind of test to see if it's better for the customers, bottom of the blah, blah, blah.

30:01 You get it out there and we know every single time what happens. All right. Well, we are going to go on to the next A.B. test. We're just going to put that production. No need to productionize it. We'll just let you let it go as is and boom, you're off to the next one and have all of your crappy ideas caught 4K cinema out there.

30:19 Can you move on to the next A.B. test? It's true. Move right on. When I was at Epic, there was a bunch of like a serious amount of code that was still unchanged from like the 70s or something. Like, I don't remember how long ago it was, but it still had Judy Faulkner at the top CEO. She wrote some stuff and you're like, dang, that's crazy.

30:43 In Cobal. While it was in mumps. Mumps right. Oh, yeah. The mumps, mumps. Let's say the coding style looked a lot different in those ones from like the 70s or 80s. They had developed some conventions that fixes, but it's like, whoa, that is insane. You just stumble on it and like, unchanged forever. Like, I'm closing this file.

31:03 I don't even want to have access to this. I know. I know. This is load bearing. Oh, man, because if there's been a bug there, there's countless customers. That are relying on the bug to exist like there's billions of dollars residing on that bug. You do not fix or touch this. You close your editor and walk away for a little bit if you accidentally end up in there.

31:28 That's funny. I've had a very similar thing happen. Whereas chasing down a bug for computers unlimited back in the day, which then later become became known as total information management system, which is like how a bunch of industrial medical like planning for equipment and shipment of, you know, auction tanks and all that.

31:42 It's like, oh, yeah. Yeah, Bisco. That was Bisco work there. And one of them is just like, oh, I have to go into synergy land. Oh, what's synergy? And it's just like this whole other language. It is old as me. It's like old as dirt. And I'm just like, I can't touch this language. I refute like this is if I touch this, I'm breaking everybody.

32:00 I don't know how to test it. I don't know what the, I don't know what the outcomes are. I don't know who's relying on this. I'm like 22 years old and they're like, just fix this bug from the 80s. And I'm like, no, I'm not touching it. I did a really bad one at Rad Game Tools. Like definitely the worst thing I've ever done in my career by far is a, so we didn't have like a build system of any kind.

32:27 Because back then there was like only like canoe make or visual studios like built in one. There was not much for cross platform build things at all. So I'm like, okay, cool. We got to get this stuff running on like Mac as well and things like that. So I'm like, I'm going to make a build utility like this famous worst decisions of all the time.

32:47 So I make this build utility called C-Deep. And it is like this, it is absolute garbage. It's basically like, it has one really kind of cool idea like I liked the idea of it, which is that you don't write separate build things. Just the, it reads the comments of your source code and that is how it does the build.

33:12 So sweet. Sweet. He doesn't know Tom's a genius. Okay. He doesn't know Tom's a genius. Sorry. We'll go over it. We should go over Tom's a genius in a separate standup episode. Okay. I don't know about to ask who Tom's a genius. Yeah. It's too long. But we will do a full episode on Tom's a genius. Next week.

33:30 Yeah. So I'm just like, this is great. So like, here's how it's going to work because and again, all of this stuff is like fairly new at the time. So the things I'm saying are not to regret this time. I'm thinking these are like really cool like innovative things. It's going to, you're just going to point it at one C file.

33:46 It's going to spider out from there. It's just going to like parse the includes. It's going to figure out all the things that it should depend on. It's just going to do it. So you don't have to make files. You don't have to list the dependencies. You're just going to point at the thing and go. And if you need any kind of build stuff, you just put it in the comments, right?

33:59 In the comment, you just say, like, here's the build parameters I want. It sounds nice. I mean, that, that is like a, you know, fear right? And if you don't know, yeah, if you don't know what the hell you're doing, you're like, this is going to be great, right? Yeah. I still think it could be great, Casey.

34:13 Yeah. Well, you and, you and me when I was 22 or whatever it was anyway. So I write this thing and I don't know anything about how to do the like, like it, I don't really think it through. I'm not good enough to like figure out how to do it well. So really what it ends up being is sort of like just a string replacement kind of language.

34:37 So you like, you can define stuff in this, like, in your comments, you can define something. And really all that's going to do is just replace that thing with whatever you defined it to be. That's all it's, that's all it could do. And it does this by like literally string rewriting. So all it is is like if you said, like, food is now this crap that any time it sees food in the things that's doing, it just replaces it with that.

35:04 And that's it. You get, there's no, there's nothing else. It doesn't have anything else. And the, to the extent that it has conditionals at all, they're just like, it won't rewrite the thing. That's it, right? So you reinvented pre-process or macros? Yeah. It's pre-processed macros at build time. That's what it was.

35:23 Nice. So this thing is a tremendous pile of garbage, it's horrid, right? And the worst part is it's the worst kind of horrid, it's the kind of horrid that seems good at first, right? Because you're like, the first time you point this thing at like a C program and it builds the whole thing and the dependencies all work and you're like, oh my god, this is amazing.

35:44 I just like revolutionized the build process for the world. I'm amazing, right? And you don't realize that actually what you did is just fucking destroy, like everyone's a tough to build things at this company for the next like 30 years, okay. So that, I do that. And Jeff Roberts is the guy who runs Rad Game Tools, excellent programmer, excellent, excellent programmer.

36:12 He goes like, oh cool, okay, so someone made this like little build utility that's great. I can now start modifying it to do what I want and what Jeff wants is always for things to run very efficiently. That's what he wants. So he adds like multi threading, like running, distributing it on cores, all this stuff.

36:32 He optimizes like the things that runs faster all the while no core improvements to this thing because it's still just a string placement thing. Rad ships on dozens of platforms, more platforms than any company I've ever seen. It ships on everything. Like, you know, it's like it from like Game Boy color to like, you know, Amazon's special arm server deployment, they run on everything.

36:58 So he builds this giant like all the builds scripts and stuff. He builds all of like the compile time sites, all that stuff. There's like tens of thousands of lines of CDIP code by the time he's done with it after several years. It builds all the stuff for like for the video to press all that stuff. Now to this day, I am told the rad builds still run on CDIP.

37:23 It is undebuggable. It is completely unreadable. It's huge amounts of institutional knowledge that Jeff has built up to build on every platform known to mankind and to distribute the builds. It is so insane. So like literally, I don't know what to say other than like I have created so much pain and suffering for that company just by creating this frickin terrible utility.

37:52 If I had just done a better job on that, it would have saved, I don't even want to know how many hours people have lost spending to try and to fair it while their builds don't work because of this piece of junk. Is the depth and sea depth short for depressed? It should be. It should be. Casey, you truly sound like a 10X engineer.

38:10 The truest 10X engineer makes 10X more work for everybody else. That's what I did. Yeah. Literally, the not having this tool would have been much better because then Jeff would have made, when if he ever did all of that work, he would have made it in something much less crappy like he would have just opened up a C file and made his own little build utility that wouldn't have sucked.

38:36 So it's like I basically took all of this other work that was actually useful and I made it suck by having it start with something awful that it never got rid of, right? And multiple people have had to come on and like fix crap or try desperately but there's only so much they can do because now there's this huge, unparsible gigantic like, you know, tens of thousands of lines of build knowledge about like, how do you properly do a annotated debug build of a GameCube binary from like, you know, the year 2001 or something like that, right?

39:16 And so that and no one wants to fix that. No, it's going to rewrite that. Yeah. So I basically ruined, I ruined that entire, it was awful. It's really bad. Casey, I just wanted to say this will make a lot more sense after Prime does his Tom talk but we just got to say Casey is a genius. Casey is a genius.

39:35 Yeah. That was real. Awesome. That was a real genius move. It was a real genius move. Um, okay, hey, TJ or hey, Josh, we need to put this in the beginning. TJ, where do we find, because obviously this is the extended, by the way, it's not, it's not J, it's not JD still anymore. It's obviously a J, myratory, uh, so trust me, it'll all make sense here in a moment, Casey, when you learn about JD, so it's my diesel.

39:58 Myratory domain specific language. Okay. I love it. Oh, God yeah. Yeah. Medisal, but the, uh, so a TG, we called it Dissol, we called it domain in a different language or Dill. It's a Dill, a domain inappropriate language. If you enjoy this podcast, check us out on Spotify, the standup, also on Apple podcasts.

40:20 Personally, I can't find it on Apple podcasts, but I've been told we have one there. It will be linked in the description. Uh, the name is Casey, computer-enhanced.com trash. He works at Netflix, by the way, TG, T streams, by the way, the primogen. Thank you for watching the standup.