Software Horror Stories

All right, today is another video on welcome to the stand up. We are doing dev horror scenes. Everybody is laughing at me because this is taken to Well, guess what we are going to hear some of the craziest stories T.J. Has guaranteed that trash would love his dev horror story So T.J. Why don't you start us off? Oh, okay, so here you go. I'll tell you the story real This is my first real job outside of school. I'm working at Epic the medical health records company, which is important to the story to know that it's not fortnight And I worked on an application called data link data link is basically a way that we turned like insurance claims in the medical data for you so like trash if you were traveling around and you broke your arm or something right you jiu jitsu didn't stub your toe but you were in like Florida instead right

The you would you would make a claim and then insurance company would send those claims back to like your home hospital network so that it would be like hey Your doctor needs to know that you broke your toe at some point because people just literally forget right so it's like you broke your arm You got this surgery whatever all this other stuff happens out of network. They want to know back at your home place so you can get better care Um, the problem with this is twofold first one. It's a dev horror story from before I was there sales promise that it would be done in three months Not a doable project in three months took nine months or something like this. This was before I got hired to be clear trash because I already know you're gonna say it's my fault. I wasn't there. Okay. It took three times as long as they said it was gonna take and then they had to delete the product because it didn't work.

And was putting patients in danger the first time around they literally got all the hospitals to remove it and they did a new version later. Okay, so it was off to a rocky start for round one. I get hired We're in the midst of round two. It's working like reasonable now for most cases. It's kind of working. So I imagine that there are a lot of horror stories that I didn't get to hear about round one but here's here's the the thing that happened that was uh That I know trash is gonna laugh at I know it for sure. So So we get we get these things are called patient safety escalations, right, which means somebody reported something crazy happening in a hospital because That's your your thing. It's not

Did we you know like stream the wrong movie or something? It's like did we accidentally potentially kill somebody, right? So so just to be clear before we get out there this never actually heard anybody we were we're confident that we did a whole bunch of research afterwards and anything but The thing that we found the reason that this got surfaced is a doctor was looking at a new mother's chart and was going through the chart just recently had her baby Going through all the procedures and about three quarters of the way down on the chart. He sees an external procedure done on her and what does it say circumcision The reason this happens is because when you have a baby a lot of times insurance companies charge everything to the mom's insurance claim ID

So they do a circumcision on a little baby we input the claims we say oh look it's on mommy's it's on mom's insurance ID You put that right on her chart. She gets back the week after having a baby. She's at her well child visit So it turns out so we find this out so this then spawns an entire like you know big problem because we're like holy cow we could be putting procedures on the wrong people So we're looking into everything we're like calling you know industry like we have a bunch of people on staff or industry experts and everything Obviously like this one it's not so bad it's chill it's like yeah it was obvious we fixed it and it was good The one that was much more like potentially dangerous in this scenario but also due to a bunch of other Swiss cheese factors like they actually got covered it never happened

Developers are you tired of code reviews that just say looks good to me or here's seven things I need you to change that are irrelevant to the PR Code rabbit AI reviews right inside your editor starting with VS code cursor and Windsor sorry no vim yet hit review and bam Outcomes feedback even before you open a pull request code rabbit can help flag bugs that your vibe coding with your LLM bots missed And with one easy click it can send all the context back to your favorite Tools for quick fixes line by line code rabbit offers a generous free tier with up to 50 code reviews per day grab the extension now At code rabbit dot links slash primogen dash VS code links in the description and ship cleaner code today

The other crazy one for this which is why you don't let sales drive what features you're going to do Is there's a bunch of problems that exist that you need industry experts to actually tell you is When you get a trans plant so like trash if you gave your liver to prime generally at least in the US The insurance company just puts the person that the procedure is going on if you're the donor it always puts it on the Doney what's the person who receives the recipient okay They always just puts the claims on that one so it like it literally this didn't ever actually happen But like it could have if we had had the scenario but it was literally like it would be like donate liver got liver right in a row

Or like because both of those claims would be associated with the the recipient claimant ID Because the problem is medical insurance data is not supposed to be medical data right so there you go trash The main one that I knew you would laugh at from that though is new mother gets circumcision procedure on chart Dude I don't know if I could ever I feel like working like in that industry is just those bugs are just like catastrophic I feel like right Oh yeah it's uh it's not I didn't like it that's why I left like I mean like epic was fine to work out I thought like I actually liked working there All right like in everything but dude when you have literally like multiple hundred millions of patients

And like all of their medical data is going through the system you're working on it's a very different experience than like Oops we like deleted a document on accident or like some photos got lost or something like that like everything is like potentially life or death So now I have a follow-up question yeah what's like what's your testing process before you like roll this stuff out Oh I mean I you how long do you want to go trash it's actually kind of like you know it's kind of interesting I mean it doesn't have to be like in death I'm just curious like pretty hardcore So part of it part of it that's kind of like so basically like at epic generally speaking They did they were changing this a little bit when I left but they were they were moving more towards like closer to a rolling release

Like every three months they would cut a new version of epic and they would try and get people on But like up until then they had basically like you know every three years would be like a new version of epic Right like classic enterprise software you've got epic 2014 epic 2017 But okay so for the testing thing so what we would do is we have these big versions and then you would need to if you wanted to get a package Sent back to people you had to create like this thing called a special upgrade And they were like basically patches that got applied and so we would have effectively like multiple Like main branches effectively like is they're like environments where it would be like oh this is epic 2016

With like the latest versions apply and you would have some like ability to say oh they've installed like these things or this one depends on this And it would do a whole graph and all that stuff and like actually lots of really impressive Tech like in that stack but so basically what we would have is for anything you wanted to change any change ever That ever got landed no matter how small no matter how big you would do that in like our dev environment This is just if you want to do it for like the main one So like first we'll just cover mains like our current version of epic that's coming out later You would do all your dev in this big main like shared environment it has tons of test data like loaded in like big scale

Try out all this different stuff whatever you can play around with it all these good things So you got to get approval we had a bunch of stuff for this design docs etc plot project tracking you know yada yada yada Yeah for once it's a product where you think okay it's probably a good idea yeah like most of the time you're like guys can we not do design docs I just want to try the feature but for this one Like for the example I said if we had a design doc earlier for some of these things and had like pulled in People from insurance and billing they probably would have called this out you know like it could have been a solved issue anyways So then you do all your dev in this environment we had our own whole custom source code,

Employment IDE review thing everything all of it was built in because like epic old started in the 70 so they built all this stuff all by themselves forever And so then you would get you would get here you would get past dev and they'd have some someone do code review you'd go back and forth you'd have to get it they'd have to sign off Like a literal sign it wouldn't let you merge unless you got sign off from people stuff like that right and it goes to qa1 qa1 you get a qa person from your department qa environment has a whole bunch of a bunch of different setup different Environments different things different kind of like a different deployment it's completely detached from dev they go through they do all their testing they've got a test script they've got regression tests they

Got a bunch of stuff like this that they run obviously like there's also automated testing and things that we did too but like you get a qa person to go test your thing once that qa1 goes and if they find anything you go back And forth blah blah you potentially need dev one to review again depending on how big the changes are qa can ask like yo you need changes qa1 gets finished you have code review number Two code review number two happens different developer can't be the same one is last time code reviewer or like testing code review so we did code review one qa1 code review Two completely different software developer house that makes sense like it what if the code review code review two ask for changes and then now you have to go back to the whole testing process

Yeah, if there's significant enough changes you'll just kick it back you'll kick it back to dev one. Yeah, exactly. Yes, if it's if it gets that far then you got to kick it back to dev one. What about the code reviewer one does he get in trouble because he missed code reviewer too. Maybe if it was a pattern you like the people who are code reviewing are on your like app.

So they're like on your like on your team. So like we have the data link team the data link team was like you know at various times between like five and 10 devs. So it's not like you would know if somebody's always just you know rubber stamping dev one. Get them out. Get them out trip to Karen's office put them on pip.

Can't be messing around with that kind of stuff there. And so then dev two completes now it moves into a completely separate environment qa2 a different qa reviews it. Everything checks out. It's good. Then it gets merged into basically like the main line like current branch for where all the code is and what's going on.

So that's like the process for just getting like one change done and getting into like the current version of epic and then like that would eventually get deployed out to different hospitals whenever they're ready to upgrade. That makes sense. Then though trash if this is a fix for an important bug or feature that we wanted to send back to people without them having to do whole thing.

You'd have to do this entire process again starting from a different initial branch sometimes could be auto applied but sometimes not because like maybe there's code you use that doesn't exist in the 2016 version. Now we're in the 2020 version. You go back. You got to go do that whole same process again and it gets put out basically as a new patch a special upgrade special upgrade.

37,982 or something like this right and that one could get sent back so you repeat that whole process again if you need to get it shipped. What's that average release time like weeks. I like you're saying from one PR starting to finished. Yeah, I guess I mean obviously depends on like the significance of the changes.

Yeah, I mean, it's super depends. I mean, if you so like one of the things that I did so this you'll think this is kind of funny too. Sorry, we'll just randomly talk about this and maybe we'll we can tell some other. We're right. We're. Tell us your horror story one of the days I release process. Yeah, that is a horror.

Have you ever had code take months to hit this actually actually have a tangent on the process that's going on here like this isn't that. Yeah, like right for the, you know, for the vertical that epics operating and you're like happy to hear that you do all of these things like move slow you're like OK, that's good.

So like one of the things I built a little bit before I left I actually built a like kind of scuffed version of telescope like my the Neovim fuzzy finder. It's the thing that inspired me to later try it out when I was done working there. But in not for Neovim because we didn't use Neovim I had epic studio while I was there when I was writing some of this code like we had our own IDE much was connected to all the environments and all this other stuff.

Although I was literally trying to write my own competing Neovim plug in distribution story for a different day. Now I forgot oh the telescope thing so I built a I built this like mini telescope thing to help our QA search for a bunch of like scripts and other stuff that they could run. And then I built a like way for them to test data link basically using like Excel files because the problem was all the stuff we're doing before was like this crazy manual process where you do something like load claims into the data warehouse data warehouse.

So you can wait until it ETLs back into our claims in the actual epic environment then that epic environment you have to test and do things and then right so it's like it could literally take hours or days or like some of these changes to go through. But you could force them manually like if you knew what you were doing right.

So I wrote this thing that would be like you could put in the claims in like an Excel table you wanted as like an input sheet and then like you would have an output sheet in Excel and you could say the things you wanted to see in the patient's chart. And so you could write out all those test cases and then I built the telescope like fuzzy finder thing so they could search through like hundreds or thousands of these like test cases later.

Yeah, so it literally changed things from like something that used to take multiple days for QA to test something manually into like less than an hour and automatic which was really like that part was pretty cool. And the literally do you like this to the reason that I found that I was inspired for this idea I'm just poking around in random old code that we have in this crazy language called mumps which is also a software horror story.

I can tell you about this after this and we had our own custom Excel parsing libraries that I discovered I was like well that's cool what could you do with that. It was like oh we could just make it so that QA can like write tests in Excel and then like we can we can write code to handle these things.

But just because I was poking around in some random ways and found Excel parsing libraries. Yeah, lots of stuff going on over there. Oh yeah, there is a there is a fate worse than death and that fate is using a company's custom IDE like that just feels like that just feels awful. It was better than VS code dude it's a thousand times better than VS code like no joke like it's not it was great.

It connected to every environment automatically it would like you could see diffs across environments and you could check it check out all this stuff. It was connect dude we had actual code review you could like push this thing and you could go check out the code you were in the environment automatically.

None of this dumb your environment scuff you're trying to do a dumb code review and GitHub and GitHub won't even load the entire diff and all this other like dude. Most companies like the experiences subpar using VS code compared to what they had at Epic using a 1970s program in language called mumps at a health care place which is also funny.

That's all proprietary that whole idea was proprietary. Yeah, they made it themselves. They were just right. That's a secret sauce over there. I'm going to get a job at Epic just to check this out. Yeah, I mean like mumps is a bad language like it is like it's true. They were working on a project to make TypeScript compile to mumps which was crazy and like it was working.

They just rewrote the back end where instead of emitting JavaScript it could emit mumps instead. No mumps. What the heck? Yeah, well it's so it's kind of like no sequel but scripting. Oh, yeah, yeah, yeah, I can visualize it now. I'm saying like somehow it's mostly most companies. Can you describe it?

No, it could be a different possible way than that. That's literally what it is. So it's a key value store scripting language where so you can like do business logic stuff on sparsely populated data sets. And you can like write it in the same language that you're doing all your other stuff. So like almost all the back end was in mumps.

And it's like a key value store language. Was the was the TypeScript to mumps compiler? Was it called like MRR vaccine or something like that? No, it should have been though. It's all the T.S. It's all going well. Yeah, they tried. So they never called it mumps. They you only call the language M because you don't want to keep reminding doctors and hospital people that you're your medical staff is written on mumps.

Yeah, well, dude, it's the classic programmer thing. Some programmers made this up at like Massachusetts university medical something, right? So they were like, guys, no, it'd be funny. Like we should call this programming language mumps. And then they because like the acronyms not even that good, you know, it doesn't actually line up.

The programmers knew they were working at a university hospital. They made this up and like brought pretty funny calling it mumps. I can't believe I spent my days doing this. What the hell's wrong with me? This is crazy. Going down mumps rabbit hole right now. Yeah. Trash, you know what? We can do WTF mumps.

I'll show you a bunch of crazy stuff and the language would be so funny. I will say I will say this is one of those things where I'll just be like the serious note is like a bunch of people like I can't ship productive software in the stack. Wow, whatever. Guys, Epic has made more money than you can possibly dream of.

They're going to stay private forever. It made Judy many billions of dollars. It's like crazy. Literally changed the world and it's a language. No one else in the entire universe. Right. Yeah. Like you can ship productive software in other things. Like it's just they deliver business value. They are doing something nobody else is doing.

Like so just too bad. Oh, we have to wait. Why don't I work or what? Yeah, if you're solving the business problem, then it's going to be okay. So and actually I kind of there were some things that were very unique and interesting about mumps that were actually kind of good. Like it's good for sparse data sets.

That's medical records. Like dude, medical records are sparse data sets. Yeah. Prime you and I have almost probably almost no overlapping data in our health record. You know what I'm saying, right? Like it's like, okay, we have maybe like that we got vitals taken the last times that we went to the doctor.

Yeah, nothing else is the same. Right. So just trying to do like some secret to the doctor, but you know, I live in South Dakota. We don't even believe in doctors here. Okay. When the guy who comes out to check if the cows are okay, he takes your blood pressure too. Yeah. Yeah. I just sit in line with the dairy cow.

Okay, trash. I'll come back with the presentation of WTF mumps for you. Oh my goodness. And then I'll give you the rundown. I'll give you the rundown. I know. Okay. I know. That was a long. That was a long. But we're talking about it anymore at this point. Trash, what else do you want to know about epic?

This can just be your time to ask me about that and then we'll just be done. We need a, we need a, we need a purpose that I want to hear. Three of four story. Trash, I got my case in a year ago. And so I'm just like, okay, lay it down. I want to hear it. I have a, I don't want to do it. I want to do like a round robin.

Do you have more than one? I'm assuming. That was the one that I wanted to tell was that we accidentally put circumcision on the mumps chart. I mean, I have like a list that I can go down some of them, you know, analyzing the list here. Okay. You have a list of these. Do you write them down after they happen of like this is a meme?

Well, you my career. No, these are like early in my career. Right. These are early. Well. Yeah. Okay. I got one. I got one. I got one. I'm not going to. I'm not going to say any company names. Netflix. But I will say it's. I will say it's not my current company. 100% government contracting. Okay. Are you guys ready for this?

So. Oh, yeah. So one would think you want to load test and prod. But. We we did. We did. And I'm going to tell you why. So when you see it, when you look at sometimes you look at sites and it's, you know, populated by some data that look simple, but behind the scenes, it's typically not you have like.

Different data sources and other things like processing it, you tell what what have you. I mean, it's not as simple as like when we just query one database and all that stuff is just there. And this is like, you know, 20, 20 plus databases, potentially. I'm not saying that's exactly what's happening in this case.

Well, let's just imagine that's the case. So usually when you have all these data sources and you have to kind of aggregate that you usually use some level of caching. Right. Because you don't want to hit those data sources constantly, especially if you have a high traffic site, which would be like in this case.

I don't know, but it's pretty high. 10 like tens of visitors a day or what? At least. Okay. At least six. That's more than I've ever done. So that's cool. So we wanted to obviously when you low test, you just want to, when you stress test, you just want to see how your infrastructure reacts. So we did low test in our lower environments.

But it's like that classic case of where obviously your proud environment is just not like your test environment or your staging. So you just can't get that data. And which is like literally the classic issue and like any tech company, right? So we're like, well, we need some real results. And we were conscious of like how much you can handle.

We did everything like we could to be like, okay, is this a good idea? Is this a bad idea? But it sounds like it's still going to be a bad idea. Yeah, you flip a coin and there was heads on both sides and the heads was a bad idea. So, so, so check this out. So this site, we do, we do heavy amount of caching because of the amount of requests we get a second.

Okay. And what we do is we have like we wrote like our own internal like cashbuster for like specific paths and stuff. But we only clear the cash at low peak times because when you clear that cash for like one second without much traffic, it's just like pretty catastrophic. Or whatever reason, we were testing like not like peak time because that's just almost as dumb as low testing and prod.

But we were low testing like I'd ever relatively low like active time like during work hours or whatever because people aren't naturally. I'm not going to say what kind of site it was. But for whatever reason, okay. The cash got cleared and one of these instances at the same time we low tested. And it took down our site for about 30 to 45 minutes, which is like absolutely unacceptable.

One, you can't log into the site. Two, you can't sign up to pay us money to get to the site. And like, as soon as it happened, we started seeing tweets. We started seeing things all LinkedIn. And like me, it was just me, it was like me and 12 engineers were just sitting on a call. And we're like, yo, it's da, da, da, whatever work.

He's like, nah, man. And I was like, we're fresh. And I was like, we're just kind of sat there like on a call like literally like just like we're doing now. We're kind of kind of staring at the space. And we're just like, okay. So tarry fresh again. We're just like, we're just like. So why don't I take so long?

Like, could you just restart things? Like, what went down that took so long to get back up? It's because it's so connected. It just goes so like it takes down. It's a basically dominoes a different infrastructure. Okay, microservices. I've heard of these. Okay. Yeah. Basically, basically, right? So we, when the cash clears, all of those systems downstream get overwhelmed.

And they just topple over. And not only does it take out like prod. It takes down all their systems that are dependent on the system. So it's kind of just like, it was like literally the worst thing that could possibly happen to me. We didn't get in trouble. I mean, it's got like a, but like, you know, I mean, it was at least top three worst moments of my life.

Okay. We had low testing. What were you doing testing or how does that work out? What, what were you doing that, like, that caused this? I'm still a little bit nebulous on that. So we're testing like the very fresh request for everything. So we have like, so let's just talk about we have layers of the cake.

The top most layers of layer we own, which is built on top of the caching on all the underlying pieces. So we assumed the caching was at right below our piece, right? And the top layer is not that important. Like, everything below that is like the most important part. But we had like some logic. And we wanted to just kind of like test the CPU readings and see like, how would we react to like X number of requests?

But it was like a freak accident where the cache had never been cleared. Okay. That's all I'm saying. But we're just testing like, just literally, do we have to spend many containers? Do we need about like X amount of time, right? Just to have like a steady baseline. And that's pretty much what we were looking for.

And it turned out that that was a bad idea. Like we loaded it. We loaded it. Like I said, we low tested and staging tests, but it's just not the same. It's not the same trash. The thing I wanted to ask though. You said that was more though. You said that was in your top three worst moments. I'm now very curious.

What are the top three? Yeah. Yeah. Give us another. I have another really bad one. Give us one more. Okay. This was probably my third year into professional software development, mainly just wrote PLSQL. Like all I do was just do database stuff. Oh, yeah, trash. So data warehouse life. I wrote a, I wrote like this whole pipeline that basically maps soldiers to send up for classes.

So we have like these benefits that allow soldiers to, you know, sign up for classes at these whatever universities that will accept them. And based on like their zip code or their status, whatever, they can go some certain direction. That's not important. But I built like this whole mapping system of just like stored procedures.

And the TLDR is like basically on sign up day, it's pretty much havoc. Like everyone, like everyone's like, I want, you know, you know, you know, when you start for classes, you want to get the best class. Yeah. Or, or you have the seven AM or later. Yeah, you don't want the eight AM one. You just don't want all my classes on Tuesday.

And I don't want like a two days. I'm going to take four days off so I can get a lot of work done. And then Monday night rolls around. So, but let me preface this. We, we tested the bejesus out of this thing. Um, I was wondering what to expect because people sent up for school. Um, and I also want to preface like it wasn't just like all me.

Like it was like a shared responsibility of a team. Okay. Okay. It's not like, we need to write this and they just looked at me and you're like, it's all on you, buddy. Right. But I wrote like a lot of the core, the core stuff. And I was like super into like, that's cool. Anyways. We tested the crap out of this.

We, we launch it. We go. It goes to like, it goes to it's testing phase just like, like with, with teach, right? We have like UIT code reviews. And we actually have like multiple layers of testing before it even gets to UIT, which is user acceptance testing for those who don't know. Um, so we're very confident, you know, like 99% confident this was going to go very well.

We're like in a war room on like the day it's launching. And this is like launch day is basically like, or not necessarily launch day, but the day of everyone enrolling. And we have like all these dashboards up. The database, I'm like moderating the database. Um, and initially things were going well.

And I was like, cool, like, I was kind of looking at the data. We had kind of like a whole like diagram of like how the flow should look. And we started getting like the initial ones, not too much traffic, because you know, time zone super early, but some people are like, do stuff super early. And I was like, okay, like early signups or something.

So we could see like things were being mapped correctly. But like on the main day when like, everybody was signing up for whatever reason. And the mapping just started going the wrong way. It's like, it's like when you go down a slide at a water park. And I don't know somehow that slide makes a wrong turn to the deep end.

And we just started seeing how slides work. Like it. And it. What? I was going to turn it into it's like when you're vain. It's like when you're fainting around, they make a wrong turn. And they go to. Those are my two, those are my two choices. And I chose the slide one because it was less or worse.

But okay. So the errors that happened, it's not, it's not, it's not obvious. It's not like you have error rates for things like go into the wrong thing. You kind of just have to look at the data and understand. Oh, that's kind of weird, right? Yep. So I had all these like pre-written queries just for like certain scenarios, just to like spot check the data.

And I started noticing obviously the day was wrong. I was like, okay. And then we also built our own like CRM system. And then we start the system just starting to overwhelm and take it. And people are getting pissed. Yeah. But anyways, like I wasn't there to like see the whole process unfold. But in the war room, we were just like, oh, I shit.

What the hell? Everyone's getting signed up for the wrong thing. And it's not something we could just roll back because it's kind of just. Because the stored procedures. Yeah, you know, you know how that goes. So I had to. I don't even remember. But we had to just basically accept what was happening.

You just had to accept it. And then you just had to fix it later. You know what I mean? It's like one of those. So it's kind of like classes. It's not like they're going to go up to you and be like, you got to stop sign up for classes. It's just it's just not that kind of scenario. It's basically the person that's helping the person is like, oh, that's weird.

We'll get back to you. And then that's basically what happens. So we had to just understand where it went wrong. So we could kind of reverse engineer the logic and put people in the right place. It wasn't like as difficult as you would think because, you know, these people had profiles. You knew like their PI or their information.

Okay. So you could kind of based off of like zip code and stuff. Like, okay, this is map clear because of X and X. But like at the time, you know, everyone's like, well, this is not what I sign up for. This isn't the class I want. This is the time slot I wanted. So a lot of people scheduled got messed up, right?

But it was weird. I remember sitting in the corner of the room had my feet up. I was just like very relaxed. Very calm. And then it kind of looks at me and they're just like, why is it doing that? I was like, I don't know. And I literally just had to sit there and just watch it happen the whole day. While I was trying to reverse engineer, why it's happening?

Cause like I said, we couldn't stop it. It just had to happen. You know, it's like a car coming towards you and you just have to just let it go. No, you can't stop a car coming out. You're gonna get smashed. You can't move. It's like jumping out of the way as you're going. Standing in what cement you can do.

There you go. That's in the now. Ty the way. Ty the way. Yeah. Anyway. There you go. That's in the now. Ty the way. Ty the way. Yeah. Yeah. Anyways. You're very bad at analogies. You should just tell her stories. They're way. So stories. No, keep the analogies coming. Please. I think this was like 2000, 2011 when this happened.

This was like my first. Like the team was supportive. Okay. And I feel like I was ashamed. But it was like my first experience of. Having a catastrophic failure. So you know. Lessons learned. And again, it was a team effort. Not all on me. But kind of all on me at the same time. So. You're not up in your case, buddy.

Well, you know what I said. I said, well, who tested this? Right? That's what I said. I was like, well, how did this get past testing? I did get best testing. Classic rebuttal. A really can't be really. I mean, did you also were you in charge of testing? Trash. No. I have a funny story. I have a funny story.

It's a side tangent. This project that I'm talking about. The passwords that our QA has been using. Like to be like a password sheet. That's where I learned it from. I really, I really use those passwords. I was kidding. I was kidding. I didn't think that was actually. They were so complex. Well, they're like, they're kind of complex.

But I typed them so many times because I would use the QA accounts. That they became part of like my being. And I just started using those passwords. Oh, my gosh. That's the one. Yeah. So, how much are they? That's the one. That you don't use anymore. Hibby with one. Old password. Yeah. No, I was trying to be.

Yeah. I'm not doing this. I'm doing it. We cannot leak your one password, buddy. Please, please, please, please help yourself. There's the lore for the one faster. Okay. This project. Okay. I can't believe we uncovered. We're going to have to tell Casey this when he comes out. Yeah. I got more stories too.

I got more stories. I got more stories. Yeah. Prime, let's hear one from you. And then I want to save a few for Casey. We can do it. We can run this back again. I want to do a part two where Casey gives us some too. Okay. So obviously the one that I can't do is the one where I created that really horrible bug for Netflix.

That was the, that got a name attack, the repulsive grizzly attack. I've already told that story. All right. So we already know. Okay. We already know. File core existed. More like fail core. Am I right, Chad? Fail core. All right. So I'll tell you this story. Okay. So. Thanks. I was. Doing PHP startups for a long time.

And in PHP land, at least back in the day, when you have a static variable, a static variable would be for the life of the request because every request ran in its own process. And so I kind of had this mentality that requests, you know, like static was just associated with requests for whatever reason.

You know, things happen. You know, you just are so used to a specific environment. You kind of forget the meaning of what static is. And so it just like happens in a specific way. You're saying because like, because it restarts the PHP process every time. Every request. For the people who don't need to be processed.

Right. So that's why like old PHP was so slow. And so I remember going to Netflix. And I had the kind of come up with logging for all the UI. Because we just didn't have logging. Like we had no logging for anything. And how this got started is that we started releasing these originals. And what would happen is that we need to have an original.

And we need to have its art for just like the box, the box art. And then there's also things called Aurora panels. Trash, I think you probably remember Aurora panels. Aurora panels for the tall ones. And then you also have these like ultra long boxes. And then you also had it in Korean. You also had it in, you know, Portuguese.

You also had it like so had all these like sub variants as well. So every single image for a show had literally like a hundred assets related to it. And we just had no idea what what places the world just didn't have. Those images. And so my boss is just like, can you fix that problem? And so I was like, dude, I got this.

I got a great idea. So I went out to go come up with like a really good idea. And my first swing at it is that I didn't I just forgot that static is not on a request level thing. It's just that I was so used to the PHP's at that time that I just I started writing it that way. And so that was like a whole situation that came in because for like the next day, I'm just like, why am I like I just made a request, but I'm in Brazil.

Like I know this is from me. This can't be real. And the thing is, is that mistake is really stupid and small. But the thing that really got me wasn't that wasn't that that I made that mistake. It's just like, hey, you've just been in PHP land for too long. You made a small mistake and all that. But my boss at the time, Jeff Wagner was just like, oh, was just like, this is ridiculous.

Open up the code. And I opened up the code. He's like, dude, you're very well static. And like having your boss who doesn't program point out the most obvious thing in the universe. Like, yeah, it's for the lifetime of an application. It was like by far the most embarrassing moment of my lifetime to have a non-programming person will come and be like, hey, your code's wrong.

And it's obvious. Like I can see it from here. And I don't even program. It just like it felt so it just felt so horrible. And I just remember like that was by far like the part that I was just like the most personally sad about things is that I made that mistake. And it was so it was so simple not to make that mistake.

Oh my gosh. And so my my stories are much smaller and nicer than yours. I've actually had very few very few like production problems. So I don't have a lot of stories except for the big big one. Except for the big one. Yeah, the one where you took down a big comment. Except for the really, really big one.

Yeah, because I always would ride I was always usually in charge of writing like the libraries like that. That was just a library. Yeah, so if you like trash, I'm sure at some point if you were in the UI side, you ran into like the Atlas Mantis logger for all of Groovy and all that. That was that was like my first attempt at making all the you to all the UIs have a unified ability to log.

So anything about them, which we had none. And the reason why we had none, fun side stories. I went to the the the Atlas team, which if you don't know Atlas Atlas is like a kind of it's kind of like a tag storage database where if you have an event. You can put a bunch of tags and the combinatorics of the tags is how many different types of data will be stored inside the database.

And so that way you can have like instant look up like I want to slice the data and all these different dimensions. And so you can see the last two days, a billion requests. You can see that like absolutely no time. So it's like a really cool thing. But I went to the the team and I was just like, hey, you know, I'm writing this stuff for the UI.

And I really need access to these APIs and they're like, oh, you guys had access at one point. And I was like, yeah, and they're like, we don't let the UI have access to these APIs anymore. And I'm like, why? And they're like, they spent like a million dollars in a day. And we took it away from them.

You can't play with this. You're not allowed to play with this. So he had the program things in there to be like, that's probably not a good idea to store, right? Like we had to be very careful on these things. And I think it thinks white listed just to be like, only these values, the UI is allowed is allowed to store it.

Don't even try that with me, DJ. You know, I'm going to deny list that request, DJ. So that was that was a lot of fun. My only other really big one was, I've said the story once or twice, but it was, I think it was Lady Gaga who was having a movie come out on Netflix. And there was like this big countdown billboard.

And I was in charge of rewriting the billboard and kind of getting it more into a modernized version of react where we had a little bit better, like kind of layout of everything. And I just get done with it. And we have this big countdown billboard. And they had all the studio execs and Lady Gaga and all those people getting ready for the billboard to have this like countdown.

And when it counts down, it's going to like show her movie and it's going to be so awesome and all that kind of stuff. But I just like messed up the testing and when it went down to zero, the whole page at freeze. And so like, so I just like totally broke the product. But it was only like seven people who saw it, which was just like just the execs and the people who were concerned about it.

Dude, I just broke it in front of them and I received very angry emails from that one. Oh no. Nice. The seven most important people. Yeah. We looked at the stats. It was literally like 12 like it was like such a small amount because it was you had to like reload the page within like 30 seconds of the video.

You're going to go like, so it's like such a small fraction of people that would even have the chance to see it. It was just all the people who needed to see it saw it not work. It was really just a second later. It would have just like been like, oh, yeah, it worked. It was just the hierarchy up the company from you to the top.

It was like the illuminati triangle. It was the eyeball that saw it. I was in trouble that one. So that one was very embarrassing. That's that's pretty much all my embarrassing stories are those ones. I don't really have any other embarrassing stories. I really didn't I almost never broke prod. I got like a couple that aren't like terrible, but they're kind of like, oh, now I know and I won't do that again.

So here's one here's this one's not my fault. And I hope whoever to be fair. None of them have been your fault so far trash. I've really kind of part of part of the time. It's usually not my fault. Yeah. I'm more a preface this just in case because I know some of them watch you for whose fault this is.

So not blaming you. It's a team effort. It's a team effort. But it was your fault not mine. But so this was. I'm not even going to say the year because it's going to be even more obvious if they're watching. So we were launching a new site. A site that I was not a part of writing. Stop avoiding responsibility.

I'm just kidding. Kind of. But so this was early next JS days. I didn't even know what next JS was. I think I don't even know what next JS is. Stuffed up for sell things. This is like when everyone's like, oh my god, service I rendering even though it's been a thing forever. It just became more accessible to people that didn't know what it was.

So it was like so it was like the new hotness and they wanted to use this new framework. So everything was just service I rendering and. Yeah. What that means is we basically just wrote like a million queries. Yeah. I'm not sure. I'm not sure. I'm not sure. I'm not sure. I'm not sure. I got to look up the definition every time.

Right. So we're using GraphQL. GraphQL had like some of these APIs that basically crawl your tree. Look for all the promises and then make sure all those promises are resolved before we turn some data. But that's a big problem when you have such a nested tree in every single part of that tree needs some part of data.

You're just requesting like a million times before it can actually render anything. So we're doing a brand new rework of a site, a very popular site. And we're like on this day brand new launch brand new field brand new brand. You're going to love it. We launched this site. It was a client side rendered site.

I think before that or it was better than what it was. We did it at that point. And look better. We made it look better, but did it not perform better. So we launched it. That's like the most classic rewrite of all time when it comes to exactly. It'll look cool. It'll look cool, but a complete disaster to like click and wait like five seconds before it even navigates.

Chat is saying it's balls.yoga. It is not balls.yoga. I five quoted that myself. Chat. That trash was not involved. Yes. That was a TJ vibe code special. Okay. So stop that. Stop. Yeah. You stop. Anyways, continue trash. So before we did this whole like next. Yes. We had another sister site and we use something that was called pre-render.

And if you're not familiar, pre-render is basically a bot that will crawl your site. And then cash those results for like a Google bot or something. So if you don't, you don't necessarily need to server site render anything. You can just pay for this bot or have a bot that just sends back cash results.

So we had something like that, but we switched to this new paradigm of using this framework. And we were able to handle a pretty high number of requests per second. But when we launched this site, a request per second per container went down to 10. It can only handle 10 requests per container. And we were trying to scale.

And we were trying to, we were trying to scale. We were trying to see the containers spin up as fast as they could. But we were just shedding so much traffic. But imagine, imagine going from, but imagine a container only taking 10 requests per second. Oh, chat can imagine. For a pretty simple site. It was actually very catastrophic.

And it was really damaging to like our, to team and the company. 10 requests per second is like impressively slow. Dude, it was like 10 to 15. And since we couldn't scale up enough containers fast enough, we were obviously going to shed a bunch of traffic. And I had a bunch of downstream effects. But man, it was, it was, it was literally so embarrassing.

We're serving multiple digits of RPS. It was, it was so embarrassing. And I had no idea like how, how this framework worked. And they're just like, we got to fix this now. We did, clearly it wasn't a bug that we fixed immediately. It took us like a week to actually resolve it. But it was just, it was a nightmare.

And one of the most stressful moments of our life, for sure. But just to watch it like release or all just sit in there. And just watch it. How, how was it that nobody knew is only going to serve approximately 10 requests per second? People had no idea back then about. We didn't, we didn't stress test it.

Like when you're in your staging environments, right? Okay, so it was just because the database in prod was so much bigger than the other one that like loading these made it take. I don't even, so I wasn't part of like the testing process before we released it. I was just part of the team and kind of watching it unfold.

But clearly we didn't stress test this. But I mean, the classic story is when you usually release stuff, you just have one QA engineer go into test, click around things load. And then once they load, that's kind of the end of the story. You never really think about, oh, we should buy a low test this with, you know, 50,000 RPS or something, right?

Yeah. And go from there. So it was just like a classic mishap of that. I don't think anyone, it was also because that whole paradigm was new to that to the team. Yeah, that makes sense. You're just, there's like cool SEO. This is sick. Yeah. Yeah. Yeah. Once once. I remember watching the graphs and everything's like exponentially up in the agency are like containers trying to spin up and keep up and just keep going down.

It was a disaster. Yeah. Lesson learned. Lesson learned. What was the lesson to learn trash? Don't server side render or what? What is the lesson? Don't try to query literally every piece of data to serve a simple page. Smart. Okay. That is like the modern Silicon Valley way. That's also. Yeah. But that's also classic, classic graph QL problem too.

People think that it's on the graph. I can just, they have no idea which resolvers are actually very heavy. So they're just like, oh, it's in the schema. Blah, blah, blah, blah, blah, blah. Yeah. Yeah. Yeah. Oh, even if you can get like really precise data, you still have to look it up at a table. It's not like you're getting much out when you're just like, I'm going to go to a huge table and perform a bunch of really bad queries.

But I'm only pulling out one piece of data that I'm going to use. Yeah, this is, this whole experience is really, really bad. Can we just end that episode?

Software Horror Stories

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin