Why Making a Debugger is So Hard! (ft Ryan Fleury)

0:00 All right, you know what I think we should do an intro and intro to the stand-up prime All right, you got this. Are we doing a podcast? We're doing a podcast today every Wednesday and Friday live on Twitch 11 a.m. The stand-up happens and today We have a special assortment of guests. We have teach as always say hi teach I like when you say that the stand-up happens every day But then you say it's Wednesdays and Fridays that one you didn't say that this time But you've said it the last few intros anyways. Hi. Oh actually, this is the second time I said what day it happened So guess what last I know what every feel or be technical. Hi, bring it back bring it back from the last episode stand-up Which is streamed it every day Wednesday at 11 and Friday at 11 a.m. You literally said it happens every day

0:47 Every day and stays in Fridays, which so what we have learned today This is insane what we've learned today is that when prime does actually do the intro correctly teach will then immediately do real torpedoes it he torpedoes it What's up? This was our one chance the plane was like leaving the runway and you were like I'm gonna be the chicken that flies into the engine as he hears my question Is this called the stand-up or is it called the well-run meeting good point?

1:14 Yeah, we also have with us Casey as usual say hi Casey. Hi Casey. I knew one of you I knew one of you sons bitches would do that and a very special guest today, which is Ryan is it flurry? That's perfect actually I knew it was French. I just I didn't know how like what you got to do with that You know, anyways say hello Ryan flurry. Hello, Ryan flurry Nice. Oh, that was wonderful. Thank you And today we have a very special topic. It's one of those evergreen topics We're gonna be talking about debuggers and since I'm a print F to bugger TJ I don't know what he is Casey definitely a debugger debugger Ryan flurry is the one who writes the debugger So we figured he'd be the best single person to bring on and help us understand all the buggers Ryan

2:10 I have a very important question to kick this off is print F to bugging an abomination towards man or is it a perfectly viable strategy? Well, yeah, I think so the first thing I would probably say to introduce someone to the topic if they were just If they were just started if they had just started programming and they're like they figured out how to insert log statements into the program to sort of see what happened The first thing I would say is that there's no dichotomy between that and what a debugger is offering like there's no dichotomy between logging and what a Debug like the information that the debugger is actually getting the only the only variable is is what work you actually had to do in order to get a log of

2:47 Information out of your program and so the first thing I would describe about a debugger is that it's collecting a log of information from your program in a sense It's sort of it's just another program on your computer that's running and it's running alongside your your actual program the program you care about that you're working on And it's saying they're collecting information from your program and it's really in a log like fashion so Now there's there are technical constraints about that because obviously the debugger has to collect information without modifying the actual program because it's running with it live and collecting information So there are limited ways in which it can modify the program some of that you know varies depending on how well the kernel facilitates that process and stuff like that but

3:29 The point is it's sort of it's collecting the same kind of information at the limit when debuggers Can't go too far. You do have to go back into your program and say okay now I'm gonna print F here and here and here and here or like generate a log or or You know plenty of other versions of that exist, but so it's not that like Team debugger versus team logging. It's very much like that there are no People should kind of remove that boundary from their heads because The degree to which debuggers could theoretically do logging is is I think more extreme than people Maybe if they're just getting used to the idea of a debugger. They might not realize so So it's not an abomination, although I think it's a lot slower than then the tools that debuggers do offer you in a lot of cases. That's that's how I'd put it

4:16 This is hard for my American mind because I only know red versus blue Like really this is really difficult for me to have some sort of new on-stans are well I mean if you think oh, sorry go ahead run I was gonna say you can re-slice it so that red is the people who think that there's a red versus blue And then or or you know vice versa. So you're on team red which knows that there's no red versus blue But blue is all the people who do and then it's just another red versus blue problem because I'm American too So that's that's how you yeah, okay, that's how you do that This makes me want to like light up a barbecue and eat apple pie. Yeah, exactly Great the car is light. Oh, I would never I Would never

4:58 Oldie the finest micro-brew All right, someone has to have a real question here to kick this bad boy off Well, I would go I would keep going on that front actually Hey, teach just finished the front end. I'm gonna get the back in in JS. Could you give me the database? That's node problem. You got a boss That was a mess without type safety. I re-wrote our back end in TypeScript. Could you fix up the database?

5:22 I'm gonna fix the ES Lint errors not any problem for me. I got this Dang, AI is so hot right now and my professors helping us rewrite our back end in Python. Could you get a database up for me? Certainly, I would love to assist you. You're correct. Let's delve into this Hey, teach Python scaled worst in TypeScript. We're gonna need go. Could you give me a DB?

5:46 Fast predictable boring. I'm ready to go Hey, teacher. We're sharing a lot of features like that. I know maybe too many features. I'm right. They don't need both of us We need to slow down, but we need to look smart. Are you thinking what I'm thinking? You don't want your tools to pick your sack. Neon does serverless postgres. You choose the rest get your free plan at neon.com So there are some issues like one of the reasons why I think people like printf debugging and say I don't want to use a debugger I don't see the point of it. There are some valid like I think those come from a valid place But they're not actually describing the problem correctly. So the valid place they're coming from is the fact that the buggers often suck

6:43 Okay, so like yes like I have complained about Linux debuggers many times And that isn't comparison to Windows debuggers which also Usually kind of suck like Linux is like ultra suck debugging, but Windows debuggers historically have also been pretty darn crappy And so you know one of the things That's important to recognize is that debuggers could give you your printf debugging way faster than you actually writing your printf debugging If they weren't sucking so bad, but oftentimes they are sucking so When I am on Linux, for example, I use printf debugging a lot more All I heard all we need is Casey's saying I use printf debugging don't give the context. Okay, sorry So and the reason for that is because the debuggers are so bad if I'm on Windows and I have something where I can do my data collection

7:38 Really efficiently because there's good debuggers largely thanks to Ryan now, but also thanks to the George Menhorn the guy wrote remedy bg Which is a bugger that I've been using previously When you have a debugger whose user interface and design is set up to get you the information that you need very quickly It will beat the speed of printf debugging because the amount of time it takes you to go add those printf lines Recompile the program rerun the program See where it print out. Oh, then then go oh wait. I should have a few more printfs and then you go back in there that speed you can Be way faster than that to collect the exact same information with the debugger if the debugger is actually good

8:16 Now if the debugger isn't good it won't beat that speed and then I am right there with everyone who's like why what I use the bar Yes, like you're right because the debugger is too freaking slow and so part of like what Ryan's project is with the rad debugger For example is to stop having that choice around it should just be a given that the debugger will collect the data for you way faster than you could have collected it by instrumentation But the sad reality is that a lot of times currently it isn't especially on Linux, but also for certain circumstances on windows as well Thank you Ryan for actually tackling this problem for us So when I say I use printf debugging that's the exact same thing as saying I use arch by the way

8:56 Oh It's like very I wouldn't necessarily go that far. I wouldn't necessarily go that far. I wouldn't go that far All right, so Ryan are you actually trying to provide a solution? That's linicite in linic sites when let me see. Yeah. Yeah, that's like so Linusations Linusations sorry Linusations right So yeah, I mean it's it's actually like and if you go back and look at I think it's on web archive at this point But if you go back and look at the rad debuggers like original Like pitch page where they actually described the project that was one of like the primary ambitions of the project all the way back then I think this would be like 2013 or something and I was nowhere near the project I was in high school

9:39 So I was like not even working on the project but all the way back then if you go back and look at what the original ambitions of the project where it's like Let's get a viable debugging solution on linics and so that's still part of the ambition and that's actually like I'm starting to work on that not quite full-time because there's a lot of stuff to fix and everything I mean with the debugger in general there's like hundreds of possible problems You could work on at any point in time Um, but I'm starting at this point. It's gotten stable enough on windows that I'm starting to do more of the linics port work And so I have like the UI running and I'm starting to do process control and we have dwarf debug info parsing and stuff like that

10:14 So yeah, it's actually going to happen like it'll show up at some point soon before Before we go further I think there's a lot of people who don't know anything about rad debugger or Ryan or other stuff I know we jumped right into the print def controversy We wanted the hot tape, but maybe we can just take a sec. You can walk us back a little bit You can talk about uh, maybe rad debugger and give people some info of what's going on Casey's mentioned it like a thousand times on the channel So I think we got to we got to tell people what it is, you know, and is it epic game?

10:45 It's the bugger now or is it still rad debugger? It's it's called rad debugger But it that's sort of just historical Epic bought rad game tools back actually shortly before I joined so I was never technically a part of like rad proper Um, but but epic owns it now, but it's still it's still called the rad debugger because I don't know that's just the That's what they just I don't know nobody's told me that it should be Renamed so I haven't renamed it But oh, but yeah, I mean basically like the story of the project is Um, it started with what I said like it they wanted a viable to my knowledge I was they I wasn't even near the company at this point so I was there if you want the actual And then I can say what happened after yeah

11:31 So what happened specifically was and this will this will just show you Remember the idea of the steam like the steam PC like steam Linux Do you remember this way way back like way way back in like the early 2010s They were like gonna ship the steam Linux set top box thing and it never really materialized like they sort some ihv's kind of did but it never became a thing And then like fast forward to today and they redid like sort of a similar plan to that and with similar people Like I think Pierre Lou and those guys were same. I think it's like similar team Did do the steam deck and that has hit like that worked and they are going with it But way back there was this sort of idea that they were going to do a steam Linux distribution

12:15 This is where it sort of actually started Because obviously rad is a does a lot of cross platform stuff They have to ship on Linux as well as Mac as well as XXXXX like in fact when I worked at rad It was dizzying the platforms we support like I'd build you know the character animation system And it's just a laundry list of platforms It's like 14 long or something like that So that was where I learned cross platform programming like the hardest way right I had a little bit experienced before but that was the big one And so rad and valve were always historically very close like Jeff and Gabe So the two heads of the companies used to have lunch pretty regularly So they were never like at each other's throats at all

12:55 They were not competitors really or never thought themselves that way because they didn't really operate in some or business spaces So they were actually fairly close and often times they would even share employees sometimes So there would be times like for example on the VR projects where some of the rad people were still technically rad employees But like valve would pay their salaries and they were at valve. There was all kinds of stuff like that So it was kind of closer than people know about because it's you know There's not like this is secret or anything but just like you know It doesn't get written about or no one cares or whatever So because that was a very like closely You know, they talked a lot

13:32 There was this idea because valve really wanted to do Linux stuff and there isn't great debugging tools that game developers are used to and again We need we also need a lot of specialized debugging tools for debugging things like 3D geometry and this sort of stuff So we need things like GPU intervention stuff and we need things like to be able to visualize vectors And all this other stuff that you might not need so we need we need ways to add that too So how are we going to get this stuff?

13:56 And so what gave and Jeff decide is they're going to do a debugger project to get a good debugger going on Linux And valve's going to fund some of it and rad's going to fund some of it And it's going to happen at the rad offices so at they expanded the rad offices They added there's like a new like area that got added on They like grew and took over a little bit more of the floor I so people seem to like when I go on ridiculous story time So I'm just telling the whole story if this sucks, you could stop me at any point you edited out great. We love it Okay, yeah, so they expanded on YouTube if you love this. Yeah, I love the BC and I can't wait for the ad coming up So I'm yeah, Ryan will take the ad we're gonna hand it off and then Ryan's gonna do the ad

14:37 Um, and so they had some people there But it was always hard to get a team of people who actually would like get it to the point like where it needed to be and Like largely that was because Debuggers are kind of I guess if I had to say I have I described them They straddle a really uncomfortable boundary You know these nuts jokes aside the straddle is very uncomfortable for a lot of programmers if they're have to tackle it sort of like It holistically might be the way Um or or head like like head on Let's say if they have to come at it from the front Then this straddle is very difficult. I just have to say KC You've been hanging out with us a lot and I love this I feel like I'm allowed to say these sorts of things here. So I do yes, it's a safe space. It's a safe space

15:31 so What I mean by that is it's a project that crosses between UI sort of like I need to know how to create visualizations and interactive things that show a lot of information in highly structured ways If I had to rank programs by what is the hardest kind to write the front end for it's actually Debugger It is literally the hardest problem that I know of if you wanted to see if a UI Programmer is actually great or just kind of mediocre Have them try to do a debugger and see what you get out of it, right?

16:05 Because it is literally I do not know of a harder problem that you would have to solve Uh, other than than debugging really, right? Ryan agrees wholeheartedly. He's like yes, one hundred Yeah, totally I mean maybe you could argue that like like 3d header like blender because it integrates the 3d part like like Those are the two that come to mind and they each have kind of different stress points Ryan decided to do both by adding 3d visualization to the debugger. So that's you know where he's at But you have the hardest UI problem or close to the hardest UI problem on one side and then you have like System level process control register inspection module loading and disassembly on the other side

16:47 Which is all ultra low level right? So when you put those two together It's really hard to find a group of people who can actually go from zero to debugger it's really really difficult and The project never really got off the ground because it just didn't have the right mix of people to actually make that happen I'll spare the gory details of that But Can I jump in for one quick second As you wise your show prime no, it's it's our it's our show, but uh, oh my god I'm feeling the love over here. Thank you. Oh one reason why I've historically keep Using VIM and then integrating dap to debugger and after protocol and then Not using it is because the VIM UI just does not feel currently kind of set up for debugger because I do believe

17:35 The bugger uis a very visual experience. It is not like a yes It's not just a separate buffers and text. There's more to it in every time. It just feels very clunky So I agree with you uis are like super duper hard to be able to build Yes And and debugger uis because they have to marshal so much information and the entire value of the debugger Is can it present is it can it allow you to edit that information display? Yes, and update and present that information display As smoothly and as quickly as possible because there are plenty debuggers Wind DBG would be a classic example of a debugger that has all the features under the hood And is largely completely unusable 99% of the time like you will only fire it up if you need a feature

18:17 That only it has like time to have a debugging Otherwise is completely useless because the amount of time it will take you to actually set it up to do the debugging you want You could have just done your print up right like it's it's the canonical example of what we were talking about at the beginning Where it's like anyone who was given only wind DBG and was like I pretty much only spread it very well I'm like yep me too yep like absolutely right so um So anyway the project kind of languished as a result of this so really not much Actually usable work happened between you know that that when it's actually it started in their early 2010s And later now I tried to hire Ryan full disclosure Out of college

18:55 Like not when he finished college, but while I was in college I was like hey Do you want to come work on this? Um, and he turned us down so that didn't work out uh, so eventually Eventually and to come on the pod to air the dirty laundry. Yeah, and suck it Ryan. We hate your guts. Yeah I came later. I don't know what to I love it. No, I was so I was so delighted I was super happy, but I was so unhappy then I'm like damn it. All right. I should have been more aggressive, but anyway um So we tried to hire Ryan and that didn't work out but Eventually it does work out and not only do we get Ryan, but we got Nick and we also got Alan Webster So like three people I put and also I mean I guess Martins never really went on that project

19:44 He was another person I tried to hire and succeeded So so uh out of those those four people that I tried to hire we ended up getting all four And through some combination of them then they were basically able to reboot and successfully complete this project Uh, and uh, now it's like it's a totally useless debugger already even though technically it's only an alpha And they really haven't been working on it that long because even the project was very long most of it was not actually usable So anyway Ryan can now pick it up from that sort of modern era. How did it happen?

20:17 Yeah, so I so I came on board in 2021 and I think Alan I don't remember when Nick joined, but I think maybe Alan was slightly before Nick But Alan and Nick showed up and they were working on Alan had inherited the project from uh, Juan Chun I believe and he so he had a bunch of stuff and it's The thing about debugger is that there's like a thousand problems like you have to know how to parse debug info formats Everything that case he was talking about you know need to know how to do process control reading registers even is like more work than it might seem like it would be And so they have all the binary too like you have to be like catch the binary for breaks and data break points and all this crap

20:53 It's cute. Yep. Yeah, like even how to implement like line stepping is already like that's a that's a deep rabbit hole already so There was a ton of work done in that direction and then um once Juan had left Alan sort of Alan and Nick worked together on a bunch of the problems around I think Alan was focusing mostly on the debugger stuff Nick was working on Nick is really good at like reverse engineering like How the pdb format stores certain things or how do you do unwinding?

21:23 You know in this particular case or all that sort of family of problems Nick is really good at kind of drilling deep and figuring out how that actually works So he was kind of working on that stuff Alan was trying to Think like work through the problem of like how do you actually build like um How do you actually build an architecture for the actual debugger sort of engine if if you will um and then I joined and my Roll initially was sort of like I was helping with a bunch of the debugger info parsing stuff But I was going to build the UI like that was the end goal like I would be on the UI like the front-end side Alan would be doing the engine Nick would be doing like debugger to perform at parsing unwinding like figuring that stuff out

22:00 um And for a while we were just working on debug info parsing so that was 2021 and Maybe six months in or eight months in I built like I switched over to building like a little prototype debugger Just so I could like familiarize myself with the problem space That has been like thrown away So that's not that's no longer in the code base But it was just like a really crappy little like let me figure out how you even do things like stepping like how do you actually Specify how a step is even supposed to happen Um and and then um Alan worked the last thing he worked on before uh he left was the we also have a custom debugging for format So both pdb and dwarf one of the so one of the problems you have to deal with with debugger

22:45 So especially one that tries to ship to both windows and linux is you have to deal with like the Situation of the different toolchains one of the realities of that situation is that these two different toolchains use completely different Debug info formats both of which are extremely complicated Um and but and they're very different in how they actually organize and slice information and so um We had kind of always talked at least for I'm sure they talked before I even got there But uh while I was there also we were talking about like this debug info format problem is really difficult to deal with actually And so if we could actually move to our own format instead and and if you start with pdb and dwarf you can convert to our format

23:24 And then build the debugger on that and then it works everywhere you don't have to deal with like an abstraction layer For example over over incrementally parsing dwarf or pdb it gets really nasty really quick um We tried to do that and then it was like it turned out to be really complicated and so uh The last thing Alan did was he designed the first version of that debug info format We should mention Alan and that's not there anymore So that's why it's like the last thing he did because he left right shortly after that. Yeah. Yeah. Yeah. Yeah. So um And then at that point once he left I I was building the UI for for the debugger And then I just sort of took over the pieces that he had he had written and I either uh

24:00 I either took them over re re-wrote them or they kind of like you know dissipated because um You know, it's just as the as the system evolves that there was a layer that he wrote that I wasn't familiar with I was like I don't I'm not really sure what to do with this. I'll rewrite that part of whatever So um Understanding your replacement Yes exactly So I kind of at that point I took over most of the debugger stuff Nick Nick actually works on a linker. This is another very deep rabbit hole we could get into So Nick works mostly on the linker, but he still does a bunch of like the like he built the dwarf Format to our debugger for format called RDI he built that converter recently um It's still it's still early but he did like a first pass on it

24:41 Um, so he's working on problems that overlap with the debugger But he he spent most of his time over the past like a year or two maybe a year and a half on the linker And then I've just been on the debugger so and the I guess that's that's the story of the project What it actually is is it's a graphical debugger that you can debug your Programs with so one thing one thing that I think people so one of the things that I was literally complaining about on twitter Just recently was people are like well game programmers just don't understand all like the real problems that banking people face or something like this Which is one of the most absurd things I've ever heard and I'm really sick of it But like just to underscore how like much that kind of thing is ridiculous

25:23 I have not ever actually heard of any other project besides fortnight that hit the limit of pdb Yeah, so the literal output of debug information on windows fortnight got so large as a project It could no longer be built because the debug info could not be stored I just want to there's a file limit size right that yes exists on windows So if you know it's not it's not a file limit size on windows It's a file limit size in in pdb The way that one of the sections one of the sex 32 bit yeah, they use 32 bit offsets actually they've signed 32 bit offsets Into like it's like the one of the type information sections And so once you get enough types and especially once you're doing like heavy c++ template metaprogramming or lots of lambda functions

26:12 Those generate types every single one of those generates types And so you get this just a massive expansion of type information Yeah, and then you run out of that space like pretty quick one. So JP Morgan call me up and let me know when you had to write your own Linker because you exceeded the pdb file size. Oh wait I was wondering though like Ryan which collab was it that pushed them over the limit was like star wars or like Which one was That's a good question, you know, oh my god, it might have been all the Star Wars characters. Oops. It's a Pokemon Collapse, you know, we gotta make sure it's got all the Pokemon You know, I think it might have been the snoop dog skins like when they added the snoop dog, you know

26:50 Doggy dog skins. Yeah, that that was like this is too much It was when they stored all the Darth Vader voice lines that got them Yeah, yeah, Sabrina Carpenter lyrics. They don't get compressed very well in the pdb So the problem is like that just and it was a problem and it's fine But most of the songs like six like six different words. They just don't compress very well Because they're so creative teach right? Yeah They're practically white noise. They're so creative. Yeah, they want to they wanted to compute them at compile time too So each word was a template instantiation and so that's why there's so many types. Yeah, that makes sense That makes sense. So the long story short There are some pretty cool things about this that if you read between the lines are there that's really awesome

27:35 And I think we'll be very very interesting to people going forwards One is rad it did release the linker Which means that now if you want to bypass the entire crappy like Microsoft and or clang linking scenario on windows at least And eventually presumably linux you can't you can have a much faster linker that produces rdi Is it rdi is the name of the file type or something? Yeah, directly So you can have that debug info which is more efficient to use and much more like well documented and and like this is easy You know, it's easy to kind of get into that's pretty awesome But too those rdi is also now there's like this convenient sort of thing you can look at where it's like here's a code base

28:16 If you would like to access the debug information for your own programs to write your own instrumentation tools Your own inspection tools your own metaprogramming tools whatever you want Now rad has done that for you or epic has done that for you right And so This this project has a lot more cool stuff in it than just the debugger Because they chose to tackle a bunch of other hard problems at the same time and succeeded I got a question for you Ryan As someone who also worked at epic, but a different one How how often like on a weekly basis After you tell someone you work at epic do you get whoa bro, that's epic every time like because I feel like it's you know Like guys come on. I'm everyone said that joke already you can relax. Yeah

28:59 It definitely definitely happen. It's funny because I was wearing an epic shirt to To a hospital at some point Fixed my chart problems really fast. Yeah, yeah, and like it was like it was one particular shirt design Where you could see epic very clearly from a distance, but you couldn't see the games part very clearly And so they're like oh epic. Yeah, we use epic here. I was like don't tell me they're putting on real engine like It's a brand new therapy We put them on fortnight. Yeah, it's like the days go so fast. Oh before you know what you're healed You know it took six months, but all the surgeons are like flossing in the room like you're like doing stuff And you're like what is happening and they're like I said a mode sorry

29:46 They always do after a surgery. Yeah, yeah, yeah, new heart new heart new heart new heart closed up that valve All right, so I actually have I have a pretty serious question when it sets you do work at epic. Do you get free fortnight? Uh, isn't fortnight already free Yes, the Sabrina Carpenter skin. Can you get that over free is what he's asking Oh, I have a friend. Okay, I see I see I haven't tried but I can I can I can check for you. Oh, I'll let you know at my epic They had my favorite phrase which was consider your salary a stipend for that like whenever people like are you Why is it why isn't gym membership included in our paycheck right because like that's the big tech man to do is like right

30:38 We're gonna micromanage your life and give you a hundred extra dollars a month to pay for like a gym right And I loved Judy Judy on stage like 12,000 people answered one time like consider your salary a stipend for that And I love that so much. I'm like yes Yeah, what do you what do you think we are some kind of health care company?

30:58 Well, they they did give really good health insurance. I missed that after I left they were you know, but yes But yeah, they probably just hack into like this. It's running all the systems It's just they set all their employees to zero dollars every time right? Yeah, it's like we're providing health care Zero. Yeah. Oh, yeah, the other thing about the epic misunderstanding TJ is not so you probably don't have this side of the problem So they're like oh you work at epic and it's like it's like no no no not that epic the one who makes like unreal And they're like oh, that's unreal. So there's like to Oh, that's a back. Yeah, exactly. So it gets it that gets rough But I just had the look of like intense sadness when they found out I didn't make fortnight

31:42 Like the whole thing they thought they were going to be able to connect with me on like I'm a software developer I work at epic and they're like oh cool Something interesting and I'm like no Electronic medical health records and I work in an insurance and billing section They're like well, that's that that's kind of cool, too Like almost like making the world's biggest video game. It's almost there What language do you write in oh mumps the conversation's over I didn't know what a programming in college. What is it C++ C? Oh mumps. Oh, okay. What the hell is mumps?

32:24 That's the language that they use at at epic. Oh, that's right. Casey. You weren't there for when we tell the hell is mumps That's a disease Why is he helps your company programming in a language this named after disease? What the hell is happening? Casey, but this is like why do people why do programmers think of dumb names because they're stupid what is like tell me no I want to know what mumps is though now. It stays for like Massachusetts University Multi-purpose system or like something like it's like some they made it at Some hospital in like 70s or something like this. I don't remember exactly. All right, and so they don't call it mumps anymore for exactly They call it m But it's it's not better. They just call it m that's very confusing

33:06 Well, yes, I just James burn them out like it's like. Oh, that's cute and that's m They also skip like D E F G like they start to see it Yes, but guys there's trade-offs the option is presenting the doctors with this was built with mumps or this was built with m Okay, they they're gonna say something find sometimes to say intersystems cashier because that's one of the people that provided I would just like to point out that basically all of the D language stands out there are now furious at prime for saying they skipped D Oh, that was that's gonna hurt some people Hey, one of them wow, seven of you in the comments. Let us know that you've written a lot of me. Oh my god Oh, we have a collection hand manual memory management. Oh, but you don't have users. Okay, so

33:53 Hey, what's one thing that both C++ and Rust users can agree on Right Oh my god you guys all right, I'm not involved in this This is nothing. I don't want to I don't want to have a fight with D people We're cutting that out. We're putting the whole thing in here and we're putting in a U laughing We're just gonna splice you out of context. Oh, yes, I should never have let you guys have edit You've never been able to edit you actually know anything about D Me I do yeah, yeah, you can just say a little bit a very little bit You can just say that if you need to what Casey, but explain this then if you're such a big fan of D Then why is it not D moratory?

34:33 I'm not a fan of D. I'm just saying like Oh, we got him. He's not a fan of D D moratory for that, you know, I'm not gonna lie to you Ryan. You've been a suspiciously quiet about Udle Could you please explain why you haven't talked about Udle once yet? What? I don't know Yeah, I don't I don't work on this Obviously you'd use the compression in all of your debugging so I would assume so at least I I don't actually like the only thing I use like one header file that that Jeff gave me For compressing, but I don't even think it's any of the Udle stuff. It's just so I'm I'm very yeah, I'm very Udle adjacent If you will do you want to talk about Udle prime?

35:24 They would come on and probably one of them might I don't know if you want to talk about Udle But I just wanted to ask an Udle question at some point. Okay. It was very important to me because it's such a funny name Okay, we'll do an Udle stream next time this time though. Can we talk about debugger information stuff?

35:39 Because I'd like to talk about that. I actually will can I do an a real follow-up question? You said something Ryan inside your talk, which is how do steps work and I thought the same thing I'm like how to magnets work how to steps work and so can you explain? Why that's such a like a tremendous problem Yeah, so so I don't know what a magnet is exactly Maybe no one knows you don't have to answer that one. It's not possible This we don't want this to become a Christian stream right now, so please don't answer that question They don't know the reference Brian. They don't know the reference. No, he does know the reference. That's why he said that No, I don't think he does the reference I don't think I know it's work. He doesn't know that's fine. You just chat move on. Okay. Got it

36:31 Okay, I think that that was the perfect response if you did know though that was like so good because we don't know how magnets work I mean no one does right? That's the whole point. Yes. Thank you Casey. I see All right, okay, so I can explain how to get back to this how do steps work? Okay, so stepping so line stepping is Okay, um, I guess the first thing to understand about the problem of stepping is sort of how How a debugger interacts with like another running program in like another native running program on an operating system So basically the debugger has the opportunity to like make changes to memory and make changes to registers And then it can suspend or resume certain threads like you can basically set up certain threads to run or others to not

37:19 And then it just says go basically you can just say to the operating system Okay, now you can schedule this program's threads again And then like the debugger loses all control at that point until that program Does something that would that would cause the kernel to transfer control back to the debugger And so if you think about what it actually means to like step over a line of source code um First you have to first you have the problem of like um locating where the uh Like where threads instruction pointer because they're stepping a thread over some line of source code that it's on First you have to like extract the register value that stores the instruction pointer of whatever thread you're stepping

37:57 You have to then Understand where that actually map like what source code that actually maps to Once you know that you know the range of addresses that are associated with that particular line of source code that that thread happens to be stopped on And then you have to say like um, you know, let me basically I have to produce the effect that This thread will stop executed like the program will stop execution when this thread leaves this line of source code Now a very naive algorithm you could do to do that is would would be to say there's something on CPUs where you can single step a thread which at that level it's just stepping over one single instruction And then it immediately interrupts So you can basically set a bit in one register on the CPU and you can say okay step over one instruction and then stop

38:42 Can I interject one thing real quick here for the folks at home So just to be clear for anyone who hasn't looked at this kind of thing before So when you have a line of source code a compiler is turning that into potentially many Assembly language instructions that will actually be executed by CPU so there's this mapping and not only is it a like one to potentially many mapping Meaning one line of source code could produce many assembly language instructions But also in more sophisticated builds like when you're running an optimizer Those assembly language instructions do not occur in a chunk So it's not like going from one to like three instructions in a row It could be going from one of your source lines to several instructions peppered throughout other instructions

39:24 It has been mixed with that come from other lines of source code, right? So the mapping is very messy and one of the reasons that debug information is not a super easy problem Is because the whole point of that thing that Ryan was talking about before with PDB's and dwarf like this debug info that gets produced when you build your program One of the reasons that's so messy is because that is trying to store in some way that a debugger can use Among many other things how do you go from a line of text that the user typed into them right To which assembly language instructions this CPU is going to execute which ones it caused because when you're going through an optimizer It's combining instructions. It's like removing instructions. It's reordering instructions

40:06 And so even just that one part is messy so everything's Ryan's talking about now when he's saying how do you go from one to the like just keep in your head It's a very messy mapping between those two. That's all I wanted to throw in there I had to also throw in just a quick like question to answer along the way Which is when you break pointer you hit some sort of stopping condition and you're in the middle of a line Does that ever happen as well like where you're just like I'm halfway through a line what line do you appear on?

40:28 Does that screw up debugging information that causes confusion? I'm very uh curious about that Yeah, so uh that definitely does happen and it can even happen like for example if you step out of a function call Like you're inside of a function you want to return to whoever called you Generally like you might there might still be some instructions to execute after you've actually popped from the call So if you do a step out operation You'll end up in the middle of the line. You're not just at the first address of the line's source Machine code you might be at some instruction in the middle um and so basically That's just an extra wrinkle in the problem. You don't um it doesn't really screw anything up

41:03 But you just basically have to say like this is a range of instructions and if the thread ends up anywhere in here I have to associate that with this particular line of source code So it's one detail in the mapping, but it's not um it's not like a showstopper by any means um So um So yeah, if you wanted to step over a line of source code If you wanted to like basically proceed this thread until it has ex exited this particular line of source code you One simple thing would be to single step that thread until you see it has left the line like you just keep mapping the instruction pointer every time it comes back And then if it leaves the if if it's changed the source code line that it's on then then the step is done

41:44 So that's that's a very simple algorithm Um, it also is enormously slow because remember that each one of these interrupt instructions that comes back uh or sorry each um each one of the single step fit uh after a single step has has completed Basically what's going to happen is the kernel is going to schedule that thread and it's going to say okay execute one instruction It's going to immediately trigger an interrupt the kernel will then um And switch to the kernel the kernel will say okay now This is i need to report this debug event to the debugger so it has to transition uh It has to context switch over to the actual debugger itself And so every single time you do a single step it's one of these round trips

42:23 You can think of it as between the debug the debuggy or the target process And the debugger itself and so if you've got a line that has A hundred or a thousand instruction. I mean, you know, you can make it arbitrary arbitrarily long really um You know you're you're causing a round trip every single time someone Uh every every every single instruction within that line. So that gets very slow So instead what you kind of want to do is say like i'm going to set up a bunch of there's another thing another concept that i should introduce Which is the int three instruction on x on x64 but there's equivalents elsewhere Um where you can basically put a It's a one-by instruction that you can use to overwrite any instruction basically because it's just one bite

43:06 So it's it's the minimum instruction size and you can go ahead and say i'm going to put this here And then when the thread executes this instruction it's called an int three And when it There's other kinds of interrupt codes, but int three specifically is this one by instruction that you can put anywhere in the code stream And when a thread executes that it'll immediately interrupt so it does it has the same effect sort of as the single step except It's caused by an instruction you put in the code and so then you so in order to Avoid all of these round trips what you can start doing is saying When the thread is in this line and i say step over this line of code. I'm going to basically set up um

43:41 An int three it's called an int three. It's also called a trap is what people call it So what you can basically do is start putting these traps anywhere you think the thread might go Within this line now if it's pure linear code like it's going to the do a bunch of mobs and then an ad and then you know Whatever very simple linear code There's only one trap you actually have to place which is at the beginning of the next line But it starts to get more complicated if you for example um if inside of that line of source code you've got a jump instruction or a call then you have to start thinking about like Where where all the places this thread might actually go to And so then you might have to start doing things like okay. Let's disassemble the line

44:20 Let's figure out like where the control flow will lead to in this line And let's go place traps in all of those places And then it gets it gets a little bit more complicated too because um for example if you're in a recursive function call Like uh, so two variants of stepping access for people who haven't used debuggers. There's step over and step into There's a lot of other step steps you might imagine doing but those are the two like common ones and step out step over will um Well, basically uh, it won't follow call instructions. So if you do a step into your sort of stop at a function call And you want to go into the function call and see what what's actually happening inside of the function call

44:56 With a step over you don't care about what's going on inside the function So you just want to step to the next line and have whatever happens inside the function happen um Now if you wanted to do a step over You could imagine um If you are in a function that is recursive and remember a debugger has to deal with any code that someone could write and feed into it Like it doesn't it has to kind of handle all these cases gracefully. There's lots of recursive code that exists um So if you're stopped at one particular line and it's a recursive call into the same function It gets more tricky because if you just place a trap at the next line and then you try to step over You're you're going to hit that same trap address

45:37 But in one layer deeper in the call stack. So you basically have descent into the next call Hit the same trap, but it wasn't actually a step over because the step over It didn't step over the line It didn't step over all of the call's work that happened Now you're one layer deeper in the call stack. So there's no way anymore to actually step over That line of code. So you have to get tricky about like Okay, have I Has my stack pointer changed when do I update which value of the of the stack pointer that I want to check against It becomes kind of basically like a little program you have to send down and say like if it hits this trap Do this and if it hits this trip trap do this and if it hits this trap then single step here and like it becomes a very complicated sort of state machine

46:18 That you actually have to implement Okay, so I can get one more. Oh, no, TJ wants it. I just got just a quick question I was just wondering like on average what your step count is every day you know, I'm saying like Because for me personally like I don't know if this makes me a debugger like a step debugger or not But like I've got my own like walking treadmill now my desk. Yeah, but I didn't know if this is kind of like What like what scenario this kind of goes on?

46:44 So I just didn't know how this related to debugger does this make me a step driven to debugger Like if I'm well, you're just you're stepping into you're not stepping over over so we're talking about stepping over right now Okay, I'm trying now. There you go. Yeah, I'm trying to stuff Harder one to do which which one is hard to how do I do the cursive treadmills? You're on an in three and you put that treadmill on a treadmill Oh, that makes so much more sense and then it's running to yeah, exactly right I got it. Okay, okay, right You can continue now. Thanks. Okay. I it's a how I okay now we have a sub question now where we our stack has been pushed TJ how long have you been waiting for that and also you have of just a black square in your

47:21 So I thought you raised it like 10 minutes ago Immediately after we started talking about step stuff. I put my I put my I put it into standing mode and I was getting ready I had to set up that scene. I did not have the video thing ready. Wow So I probably heard about five percent of what Ryan said, but I was so excited Wow So excited All right, that is some serious professional level podcasting all right All right, so when it comes to stepping and putting into all these in three Do you also have like because I don't precisely understand how jump instructions work I believe there are relative and absolute ones Does that mean you also have to update all the relative instructions around all these places that you put these in threes because

48:07 Thoretically you shifted down some amount of it's like assembly So you're not inserting Okay, I thought you were dynamically at like runtime being like okay like they're stepping next Let's actually insert some you're over you're over writing it. You're you're over That's why it's crucial that in three has a one-byte version because you're you're romping it over the instruction One of the things Ryan didn't mention because he had to mention like 8,000 things because this is incredibly complicated Is that actually when you write when you put that in three in there you have to remember which bite you overwrote Because once you are done stepping you have to go replace all of those with the bites you overwrote

48:48 So that happened as part of this process Oh my gosh, okay, so does that mean when you literally overwrite assembly and then when it hits that in You then have to put back in the correct assembly so that it when it executes the line of correctly execute line That's a that's that's that's crazy dog I don't know. It sounds like probably debuggers. It's as easy as making a regular crud app guy So Ryan do you have any idea about a banking app? Yeah, uh, yeah, I can't even imagine It's just That's like magnet level stuff right there Yeah, how do banks work actually we don't know It's actually you would have been paying attention to twitter Lori wire just tweeted how a hc works, which is literally uh ftp. So yes, all right

49:38 We do know how banks work Yes, we know that's why anyways So underlying all of this right is like this is this is also why it you may have noticed there was a little snippet of Ryan at the better software conference doing a little thing with Wukash Where he he asked him like what was the one OS feature that you would like to have that you don't have right now And Ryan said I would like to have user level in three and it's like what What is that right um and that underlies us entire thing so what happens is in the CPU right There's this thing called the interrupt table or there are things called interrupt tables in CPUs right And the reason for this is right at the beginning this is it this is in like 8086 like the original 8086 chip has this

50:26 This is not like a new thing They've been upgraded over time to have different capabilities, but generally speaking this exists for a very long time um So they have these tables because what you often want to do is you want it to have some kind of like Higher level supervisor higher levels of room term super visory uh system level I should say lower level code operating system code whatever you want to have some code that you can trigger from anywhere Instantaneously meaning you don't have to like no in advance that you were going to use this code at this time right A call is when you know you're gonna use it at this point in the code. I call this function right This is like we have no idea what we're going to do we just suddenly have to do it and one of the big reasons for this originally is like

51:13 servicing hardware So the CPU has to read something from some bus on some hardware, you know like the original keyboard Inputter whoever knows what's happening right and so what would happen is an interrupt Literally the reason it's called an interrupt in three interrupt three an interrupt would happen that stops the normal execution What the CPU was doing?

51:34 It looks in a table for what it should start doing it goes and does that and then it resumes back to wherever it was in the program It's literally what it sounds like it's an interrupt and this was used for doing things like I said like reading hardware and stuff like that because Bites would like come into the CPU that it had to like look at right now So it would stop the program, you know This is back when do you know multiple cores or anything like that right stop the program as is executing right in the middle of whatever it's doing Go service this interrupt is called servicing the interrupt right go jump to that code do it right So what an interrupt really is is it's basically the it's the same kind of thing as a function call

52:13 But it's one where you didn't know what was going to happen. It's a function call that happens out of nowhere And the reason that this is very useful for something like debugging is because one of the things that a function call will do Right is it's going to save what the instruction pointer was before it does the call So we will put the instruction pointer on to the stack right this is a call instruction We'll put the instruction pointer onto your program stack It then transfers control over to whatever the location is you're going so it changes the ip to point to whatever the function is that you wanted to call Does that and then when it goes to do a return It will pop off the stack whatever that instruction pointer was so it's basically a thing that uses your program stack your standard program stack

52:56 To track where you are in the program that's a normal function call And int can't do that because the it can't even know if there is a stack It's happening in the middle of any code there may not even be a stack frame You may not even be using the stack pointer as a stack frame at that time technically in 8086 So what it has to do is do some special work to preserve right like the ip in this way that will like we know we can preserve just what the instruction pointer is Then we're going to go into whatever we're going to look in this special table that the CPU literally has a hard coded register that sells it where it is So it's separately stored right We're going to go there look up where we should go and by the way nowadays

53:40 We also look up a bunch of information like whether we should be transferring control to like a different level like should we be dropping down to Ring zero from ring three and all this other stuff There's all bunch of crap in that table and off you go So the way debuggers are written currently because they have to be is On something like windows They are written as a user level application Which is fine because that's roughly what you want you don't want a your debugger to crash to take down your whole system Right, you don't want crowd strike happening here, right?

54:10 So you want the debugger to be running at ring three on windows so that it's running up like a normal program And if it screws up like if Ryan on like the one buggy has right if that bug if you hit that bug and it crashes your system You just want the debugger to crash you don't want your whole system to blue screen So it's good that it's doing this But the problem there is it needs to do all the things that Ryan just says it needs access to the memory It needs to be able to put in like these things like in threes that will happen And the problem is when you are at user level right the kernel doesn't want you doing stuff like messing with the interrupt table Or you know saying that you're going to be the person who gets whenever an int three happens in this on this core

54:52 That you're the one who's going to get it or whatever right it didn't want any of those things for security and stability reasons So as a result the debuggers have to do all of the work that Ryan described through an API that the kernel will then use It you know it takes your API calls it goes. Oh, I see you wanted me to Tell you when this guy hits an int three So you go you you know, I'll let you access his memory through my memory mapping API's you go you poke it in there You poke your int threes wherever you want great when the int three gets hit the kernels the one who's going to get called Not the debugger and then it's going to go through the API to let the debugger know that it happened

55:29 What this means is that now all of these things Ryan's talking about every one of these traps that gets hit is a full round trip Debuggy is running at ring three The trap gets hit the int three gets hit. It's got a drop to ring zero Wakes up the kernel the kernel goes wait. What is happening? Oh right? This is a debugged executable I know I should tell the debugger about it. Okay. Let me re-switch the task over to the debugger's context Let me wake up the debugger in ring three and resume execution of a debugger thread to handle this The debugger then goes what just happened? Oh the kernel told me I had a trap All right, let me go fix up the trap things Let me set any new traps that I need and then let me retell the kernel back at ring zero to resume the debuggy

56:13 Back we go to ring zero It then switches in the context for the debuggy back up to ring three and we resume where we were going right This is wicked frickin slow for so many reasons right and so what Ryan was talking about is hey man What if we could just set the int three to just jump to something in the actual the person who's being debugged that program The debugger could like jit compile some code that just does the trap operation that it wants to do and resets Whatever it needs or anything else right checks conditions anything it wants to We just set the int three to not do any of that don't tell the kernel don't take your in transition Just jump right to that thing do the stuff and then resume it would be so much faster

56:58 Getting rid of that like first ring tenses and back up ring transition back down ring transition back up ring transition right It gets rid of all of that So that's what he was asking for and this would be like a huge feature if there any kernel devs out there watching please add this user level interrupt handling like this would be so useful And then that thing if it actually wanted to do that huge transition thing can just have a sys call That it does called Actually transfer control to the debugger right so anyway. Yep. Hold big thing of coolness there We can't do all I got from that is horse and debuggy is slow. That's what I got I tried so hard Ryan. I tried so hard I tried so hard that's the problem. Sorry anyway

57:45 Yeah, like what what example of so at a high level what you might imagine doing in a debugger that uses That sort of where you want that kernel level feature would be If you set a breakpoint on a particular line of source code if you hit if you like run your program within the debugger You say stop here That's perfectly that's like recently fast because those those To like to transition or three or four transitions actually The um you pay that cost once and at the scale of like oh run to this line. It's not really noticeable And you wanted to come back to debugger anyway, so it's like doing what you wanted roughly right yeah So when that gets really when you actually really have to pay that cost at us to a sort of prohibitive degree in many cases

58:28 Is when you wanted to say stop at this line of source code But only if this condition is true so conditional breakpoints are like super common debugger feature And it's like kind of the first version of fancier stuff that a debugger would want to do um But this is where it gets really slow because then basically um the debugger can't like do this in Installation of code that checks the condition inside of the debuggy And so what has to happen is it hits the int3 Controls transfer back to the debugger the debugger checks the condition it evaluates some expression It like does whatever evaluation stuff it needs to do to check the condition and if that condition is false Then it goes back to the debuggy and so if you imagine you put this in the loop that's supposed to run a million times

59:09 Maybe it happens instantly Normally, but because you're paying this round trip cost every single time the thread hits that in three It's uh, it just becomes like prohibitively slow now if you're doing it once per frame like it's still worth for It's still worth it for your debuggers to have conditional breakpoints today But they're like dramatically pessimized just from this problem But if that problem were addressed you could start doing lots of other things like another thing Something like pre-neft debugging is profiling like marking uh inserting profiling markup in your program Like if you wanted to say oh, I want to sample the time here sample the time here Let me measure that build me a flame graph of this kind of of this kind of thing

59:44 That is the same problem as print debugging where it's like okay Go back to the source code insert the things that I was looking for build the program run the program Get back into the state that I wanted to be looking at and then inspect the log and then it's like Oh, I need to drill one layer deeper into this into this like flame graph here But I didn't add the markup there So let me go back to the program add the markup there and you can't do it too much You can't do it at every function for example because then you know, then the that profiling cost will start adding up Like dramatically and then you won't even be able to see what the actual performance costs were so What you what the debugger could do is like let's just do that dynamically

1:00:17 In the debugger, let's say add a add a line of markup here like a sampling line here Add a sampling line here and then it could gather that data completely dynamically Be way faster to like drill down into problems when it's really uncertain What you need to be looking at basically So so yeah, that's like the high level what you would expect as the user if you had this kernel level feature basically And so what we have to do now basically right if you want to do this kind of thing you can still do it But what you have to do now is you have to handle a crap ton more edge cases basically So what happens is if you want to you can still do effectively this by not using int3 right because remember that the important part about

1:00:58 3 is it's a one bite instruction so even if it's overwriting another one bite instruction It's okay all the code will still work exactly as it should because you only have the int3 just exactly where it needs to be And anyone who's like jumping to the bite immediately after that doing something weird or who knows what it's fine right However If we want to implement a feature like this without the assistance of int3 Or halt or any of the other instructions that go through a table and our one bite right there aren't they're very there aren't very many I think there might only be those two. I don't know if there's a third one but anyway On x86 anyway So if you want to do that without this and you know Ryan has looked at this sort of stuff

1:01:40 And this is this kind of thing people are currently doing so this is how you have to do it You have to input some kind of a jump or a call actual instruction there which takes more than one bite So you can still do everything we just said, but you have to overwrite the thing you were going to overwrite with several bites Potentially and when you do that now you run the risk of oh if those several bites Overflow and go into the next instruction that could have been instruction that someone was jumping to So now we have to do all this checking to figure out like is someone jumping into the place that we had to overwrite to and blah blah blah So now what should have been a very straightforward extension of just patching in an in three and everything works perfectly

1:02:21 And guaranteed no problems What should have been a perfect extension of that is now this huge like oh do we catch all the corner cases oh crap It doesn't work in this one place and now we got these bug reports and these guys that were doing this thing and like you know And so if the kernel could just Add this feature all of that complexity goes away and there's like I said So this is a thing that people have done. It's not like you can't do it It's just it turns what would have been a very simple problem into a very complicated problem And so that's why it would be great if kernel could do it Kernels are great at that Yes, that's why we call them kernels because they're like a kernel in a military thing where they I don't actually know

1:02:57 Yeah, keep going. Yeah, go cut it out and post dry kernel keep his armies Shoot shoot forward again Casey here. It's like the kernels are great at that. Yeah, yeah, just like it's right We're gonna take it made a beep Um, does it so are all these things you're describing effectively gone and things like JavaScript or in any Interpreted language which has a VM because then the VM can go well I can just insert and interrupt like right into the bytecode I don't even have to worry about it since I'm processing these things this does do a lot of these problems effectively because the chrome debuggers actually Shockingly a great debugger Well Okay Well, I mean you got to remember your debugging JavaScript. There's not like a lot

1:03:38 You know, it's it's I've had a lot of I've I've been not that happy with its accuracy and catching break points and things like that Actually, so I would not categorize it as a great debugger personally Really I would I would say it's some pretty mediocre debugger in my opinion You don't even program JavaScript don't even I do I've broken a lot of I already know that this is a trap This is an in three I am not I'm not getting bamboozled here Casey I have I have used that debugger a bunch. I had to use that debugger on electron even prime So I use it locally Local electron fun and that was a huge ball like that was that was great that might be different. I'm not I'm not good It's the same, but I've used both of them. Yeah

1:04:22 It has a it had I feel like it is not reliable it is not reliable a couple things Restarting and catching break points when a web page is reloaded I have found it to be fairly unreliable actually and that's a very important thing Because if you're trying to bug stuff that happens that start up right like at at load time It's just and so you know, it's it's one of those things where I didn't When you use it a bugger you get this feeling of is it solid or is it not solid right not like not the SOLD like Acredium SOLD thing not list cough substitution principle But like is this thing reliable so that if I set a break point I know for sure that this code only got executed when the break point got hit and no other time

1:05:00 And I definitely can guarantee you that was not true I think I can actually explain that one that one I think I can explain because you if you use the keyword debugger inside JavaScript you'll actually get it every single time But if you use the actual chrome debugger to do that it uses the CDP the chrome debugger protocol Okay, so it has to do some sort of async communication with your process If it loads fast enough you're going to miss that entire plane is what I'm like 99% sure that's what's happening because I've ran into a bunch of these issues Okay, working on working on that protocol So now you run into the issues prime at first yeah, she said yeah, okay It is a great one if you're if you use it correctly, right? I mean come on

1:05:35 But that you shouldn't have to write the point of a debugger to save you time if I have to go research Why this thing can't set a break point and it's like oh The thing that's supposed to set the break point you don't set break points that way You have to use this other thing that's not how you set a break point to set a break point Like then it's then that's worthless because again, I could have just unprinted Which by the way is what I do I do console.log when that kind of thing happens So this is why I say I totally understand people who like go I don't use debugger It's like yeah because when the debugger is unreliable It's not useful Yeah, for me, I just try to not write bugs in JavaScript and that seems like all right

1:06:11 Yeah, I don't have any bugs so you know that's why I don't use debugger I don't ever have any bugs. So I'm not really sure like it's cool if you have bugs I mean you could use one I guess I mean I get it. Yeah, why Prime might need one. Yeah. Yeah, that's because I always forget no mistakes It's just like if I could just add that to the end of my AI problem everything would be great See that's the problem you got to put it in the agent instruction. So no mistakes is in every one See make it secure make it good don't put it in a firebase bucket with no authentication But also if we could expound on why I don't think the current debugger is very good So going further along those lines So again in order for debugger to actually be valuable it has to do a lot more things than just stopping where I tell it to stop

1:06:54 Which already it didn't do So the other things that needs to be able to do is gather and like martial and display information to me like cleanly and quickly Right, and it also doesn't do that very well like yet kind of has a watch window That's not absolutely horrible or anything But it can't do any of the kind of stuff like I love like all the rad debugger features where you can kind of go like Hey, here's my data structures I want to turn that into tabular data So like walk this link list for me and show it to me like it was a table and like all that stuff like You can't do any of that stuff as far as I know and if you want it to do that stuff You'd basically be like programming JavaScript You know, you'd have to be like writing your own like JavaScript stuff to do that right in there

1:07:32 Which is again is taking me more time. I'd just do a console log And so I think like yeah like chrome debuggers pretty mediocre I I'm glad they built it, but it's it's not it's not in the top tier like it's not an s tier Yeah, we're by nature. Oh, we got to come back with the debugger tier list Casey We're gonna do that offline and we'll publish a video I think actually could we end with some of that?

1:07:53 Can you talk a little bit about like the cool rad debug features because like some of that stuff is really cool Most people don't know you can view like 3d models in rad debugger, right? Yeah, yeah So the 3d models like some of its early so the 3d model it's solid, but it's but it's it's I need to work on Each each visualizer is like a complicated problem that I have to dig into later and so but I do have a 3d visualizer So basically the way you can the way it works is basically with The debugger's got a watch window and in the watch window you can you can enter expressions and they show you you can expand them and all that kind of thing Um, you can also extend these expressions with like visualizer rules

1:08:34 So if you wanted to view like some like some The very common example is viewing like pixel data or bitmap data If you wanted to say for example like I have this pointer to some giant blob of data That's that's just a sprite sheet or some some you know font at atlas or something like that um What you can basically say is like Okay, I want to say that this is a bitmap and then I want to specify some extra information about like what kind of bitmap it is Like how bit how how why does it how tall is it what what pixel format is it in And the expression language like in the debugger watch window just allows you to express that kind of thing So you can say like bitmap and then the first argument to it's like a call to bitmap

1:09:16 You can say here's my pointer here's my width here's my height those are just expressions so they can come from variables in your program Um, and then you can say here's my format and then you can expand it and it's going to show you an actual Preview of of the image itself You can also like pull it out to your own tab and like see like oh like here's a full fully featured at the limit Like once I'm done with it. It'll be fully featured But you know, but it's like pull it out to to a tab and then it's like oh here's a bitmap view I can pan around I can zoom in check the pixel values and all that kind of thing um Now every visualization feature in the debugger is sort of implemented in this way

1:09:49 So uh another common one that debuggers will often support is like a memory viewer like a hex view of some memory It's exactly the same feature in the rad debugger where you can say okay memory call let me call memory Here's my pointer. Here's my size like you can say like number of columns number All kinds of parameters that you want to do for building this hex view or basically you'd span it It's just a memory view of whatever parameters you you specify to it As Casey said, there's also a geometry one So if you if you you have to specify some pieces of information Where you can say like here's an index buffer. Here's a vertex buffer. Here's how many indices. I have here so many vertices I have um, I don't know if you need to specify the number of vertices

1:10:30 I have to go look at it but you can basically say this kind of information and If you expand it you just get the 3d representation of like here's all the geometry dead that you actually Actually wanted it because this is in a watch window and it's not just like parsing a format that you dump out of your program You see it update live as you step so you actually like step over As these vertices are being built and you can see the vertices change in real time So it's it's way more useful than like dumping it and then like dumping it to an obj and then looking at like some obj viewer like this 3d model or whatever or same thing with texture data um, and so there's also a bunch of other visualization features like

1:11:09 More basic ones like viewing multi-line text For example, it's something that every debugger has to do anyways to show you source code But it's like you should just be able to apply that same exact path to like just viewing any arbitrary textual data The disassembly viewer is also implemented this way. It's just a visualizer And then there's a bunch of different rules that you can introduce to like change how it how the watch tree So when you put in like a watch expression if you expand it Um, you know, you get rows generally they're formed by looking at the members of whatever type you expand or something like that There's a bunch of ways you can customize exactly how that looks like as well

1:11:44 And that's that's sort of more like sort of what visual studio did with natvis It's a very similar kind of although it's not a next you don't specify that in xml in the right But you can do like the same kind of thing there um, and then I guess the final layer I could dig into more if Of any of that but uh the final layer on top is you can also automatically associate certain types with certain visualizer Expressions so you can go ahead and say I've got a struct. It's called bitmap It's got a pointer a width and a height and a format and I'm going to say if you evaluate this type anywhere in the program Or anywhere like if you're in the debugger and you're evaluating anything of this type Just automatically say it's a bitmap visualizer

1:12:20 So you're sort of extending type information you can put this directly in your program You can put it in the debugger configuration like really wherever you want and then anytime you evaluate that in the debugger By default it'll show you that visualization automatically So that includes hover evaluation So if you hover the bitmap variable in your program You'll see the bitmap just pop up if you put it in the watch window It'll just be the bitmap like it's you know That's that's sort of the final layer to it where you can basically extend your program's type info To be automatically associated with this visualization information now you can always go back Like if you're like I don't want to look at it as a bitmap

1:12:54 I want to I want to look at it as a struct because that's what it is you can always Reverse it there's there's a there's a way you can do that inside of the visualization like Or within the expressions syntax as well, so But yeah, there's lots and lots of stuff you can do And it's that's that's like the that's a good first run through I think It also loads fast Yeah, yeah, I mean, it's not it's not hard to beat visual studio Yeah Visual studio can't beat visual studio. So it's more difficult than maybe they think you know So I just throwing it out there. I feel like it's a cool feature is that it actually loads. Yeah Well, also like some of the things that Ryan was talking about just then are important in the way that I was trying to describe before in other words

1:13:42 It's really important to remember that a debugger having a feature doesn't matter if it takes you longer to use that feature Then print after bugging would have been defined it right or we can extend like the phrase print after bugging We're using that as shorthand because like for bitmaps you wouldn't be print after bugging But you would be doing a summer thing which is like dumping the bitmap multiple times I've literally written this code before I had read debugger where like in the in image processing code I just insert like dump image here dump image here dump image here dump image here And I look to see right like where did the problem happen?

1:14:15 And then like oh, okay, right? And so it has to be faster than that Because if it's not faster than that There was no point to the debugger having this feature And so some of the things that Ryan's talking about are way more important than they sound The ability just to view something as an as an image in your debugger isn't enough It's really crucial to have the ability to also say look when I built this program And this is a really cool feature debugger When I built this program I just included this little macro that I put that Stick some stuff in the type information automatically when I build the program that just says any time I'm looking at my struct bitmap you know structure It should automatically set it up so that I can view it as the bitmap

1:14:53 That is like one little line of code I put in and then for the rest of Anyone ever using this program inside rad debugger They'll always get it shown as a bitmap automatically with the right width and height and all that stuff right And that is a massive time savings for everyone Whereas if it just has this way of like I load a plugin and then I instantiate the plugin And I say that I want it bound is like if I'm doing that every time I have to view my bitmap It's not saving me the time that I actually need to debugger for so all of these things are wicked important That's why it's so cool like that rad debugger is taking that approach of like we actually care about saving you time Not just we cared about writing on the thing it can view an image right features can view image

1:15:34 Yeah thing on the front page that is not enough right it needs to be I can view and debug image processing faster than if I was print up That is the actual feature you want in the debugger because otherwise why we're using it Yeah, yeah, you'll see you'll see demos of that exact feature too not to not I don't want to pick on anyone But you can go look at like demo videos of their like look we can view like images inside of our debugger And in the demo video it takes like 10 seconds to load or like 30 seconds even to I can't even remember what it was It was just like it's not it's not just 30 seconds and then you can see it It's 30 seconds every time you step like it's not updating like right away

1:16:12 So it's it's just like how is that helping me like Yeah, so I also another good example of printf that's not printf exactly it would be graph is like if you if you log a bunch of Like if you printf a bunch of graph is like textual data you get like a pointer graph view You can like print out a pointer graph and then you get this graph structure that you can visualize and see like oh Is this tree wrong here? Is it here or wherever um I've been implemented this visualizer yet So it's not in the right debugger yet But it's like that's the kind of thing where I'd like to push it where it's like no you can just you can just um see your pointer graphs Update in real time as you step and and so Yeah, that kind of thing would be really nice

1:16:52 I'm we have to write in it. We have to write a real language though. I know I'm sitting here thinking the whole time like man TJ made me do elixir and now I can't change like now I want to know I Can't change languages just to use it. We have observer in elixir. We have observer. It is actually good like it actually is elixir is not a compiled language It compiled to the end stuff. Yeah, and then it runs But The observation tools are very cool because it actually is set up to observe everything across your distributed system Which is like sometimes kind of difficult they can be running on other computers and everything right like it's not so Not so simple to do but elixir has very nice Tooling around this. Well, that's kind of a bummer, but like yeah any any compiled language you could be used around it

1:17:37 Like you know, you could use rust or something, right? You don't it doesn't have to be simple. It's frustrating It's it if it's compiled and produces debug info Then it can do it so you know, you could be using it with ginger bill was on you could use it with Odin I believe just fine, right? Well-known debugger hater uh, he said it doesn't work for Odin so I doubt that he also hates package managers and alopeas as well But okay, that's just a joke from yesterday Okay, I have I haven't I know it's false too because he he literally put in special features into the Odin compiler Yeah, to work better with the random. Yeah, he was literally in chat five seconds ago saying I'm gonna go watch from the beginning

1:18:16 So that I could see what's happening. I just wanted to say I use rad debugger every day Yeah Oh, actually, oh, okay, they were joking I'm about to get a very angry email and then he's gonna watch the rest of the episode It was in his BS it was his best offer BSC come Talk he literally put it was like rad debugger and he announced that it was coming to macos, which it's not so that was back off That was interesting right So I don't know he he basically is forcing he's that this is a this is a well-known like a Gabe Newell tactic He will just announce something so that then everyone has to ship the thing because it's been said publicly right?

1:19:00 And so like he he took a page from that playbook is like well if I announce it's coming to mac It's got to come to mac That's TDD right talk driven development. Yes. Oh, yes PDD press driven development. Yeah Well, I'm very happy Ryan you came on the stand-up today to announce linux is gonna be happening in the next few weeks So That's crazy you heard it first. Yeah, here first Thank you Ryan said Tim Sweeney personally told me that we Yeah Yeah, well thanks for thanks for having me though. It was there's a lot of fun. Yeah, this is great. Thanks for Explain how complicated step stuff is It's it's so ridiculous when you think it's gonna be easy, right? It's like I just stepping over this line

1:19:50 It's like nope yep. Yeah, the recursive one really blew my mind because that actually just I mean that one has Cuz now you're like ref counting basically, right? Yeah, yeah, yeah, I thought the stack pointer thing was really clever I do you use the stack pointer effectively like save the original address at that time and then Until you return to that stack pointer. You're like I'm somewhere within The recursive brand We do have that path, but we actually figured out something cleverer for the recursive step case Cuz generally like nice try buddy nice try, but actually thought of something not stupid Okay, it's first off. Oh second off All right, so tell the tell my dumbness how you do it then

1:20:35 I remember when I was naive. I thought that might be an option Actually, you know what I want TJ to explain it really quick. Yeah, yeah Any any time a guest tries to politely say how they did something teach is just like this is an opportunity to make fun of prime Yeah, that's true for me stepping's very easy. I just do left Yeah Right, that's like what I was showing earlier. I literally did a live demo. I was like here's how I want dude. I'm stepping right now Okay, so what is the cleverer way yeah, yes, so the clever way you can do Is um instead of checking the stack pointer on every recursive like every recursion downstream of wherever you wanted to step over You would hit that trap and have to check the stack pointer

1:21:22 What you can do is and and so like recursive calls get very expensive to step over that way What you can do instead is you can basically say when you call into the recursive call or really what whenever you call into Anything because it could it could call into one function which calls back into the same point It could be one of those like ping pong recursive recursions. So anytime you call into anything I want to catch whenever that call returns back to me Um, which I don't I can't rely on the code address to do that because again It'll hit that trap first. So what I can do instead is basically turn off all the traps that I've set to to catch this step And then I'm just going to completely fake an instruction pointer on the stack so it will but went so it because when when you call it pushes the return address of the instruction pointer onto the stack

1:22:08 And then when the call completes it'll it'll jump back to that nice that call address. Yeah, yeah So it's it's super nice because then you can say after this call happens It's going to have pushed some address onto the stack which is the real return address. I'll romp it I'm going to fake that nice and then whenever they return I'll get an exception not from an int 3 But from I just x you did some rent like whatever the bogus address was we happen to use 911 So we just like we just say like So we put 911 in there is the return address and then we'll get an exception that's like oh like I don't even I can't even access that page like there's no code there. How I'm trying to access address 911 And so the debugger that means oh you've finished the call move back to the original return address and then reset all the traps

1:22:50 So fairy clever nice question. Yeah, that I have two questions. I have actually Okay, so the first one is all right get hold on. I got a pee. I have to be Hey prime nice shirt Did you guys see a shirt? It's pretty good. I didn't see a shirt. I just when he comes back you'll you'll laugh Okay, so you just ask your questions. It's okay prime can watch it watch it back later Well, can you go grab like when they edit this together grab video of prime still sitting there going like Oh, smart, right? So it looks like he's watching the whole time Well, at the tower because we use wireless mics I think one or two times he went pee with the mic still on and you could hear it in your you could hear in your ears trigger and assert

1:23:43 When I click here that's sick You just pretty good. That's courtesy of low level. Hey, can we move at least the line down dude? Oh my goodness Fry fry Oh god primie's doing it. No, that's literally like the naked gun right like that's literally a scene from the first naked gun movie Yeah, and we I had in here monitoring, you know for like our mics to make sure levels were fine and stuff So like I'm just sitting here. I'm sitting in the middle of the room and I just hear him and I'm like what again? Yeah Okay, so my first question is I What happens because I have no idea what happens in that scenario with like the return address The special return address stack does it actually is it smart enough is a CPU smart enough to go like

1:24:37 somebody Modified the IP on the actual stack. So I'm gonna like go Modify the return address stack or does the CPU always I've never looked at this before it is a CPU always check When it pops you might not know this because maybe it just works when it when it actually goes to do a ret Does it just check the return address stack and go if it didn't match Then I don't do it or something or do you know I to my knowledge it always will just take the address and jump to it So it'll it'll And I don't I don't know for sure because it did just work. Okay, we're basically like let's just phony this address And then it just went to it and we were like okay, it went to it. So then Ed says yes 100% you're right Ryan low-level TV confirms in the chat nice confirms what

1:25:26 Exactly what you said that it doesn't check Ah, okay doesn't check these they doesn't it just does a pop. I don't know it that's what it has to check So just to be clear inside a lot of CPUs now there so there's the stack as you think of the stack Meaning the actual program stack that we push like arguments and stuff on or like stack variables or whatever the crap right like Things that don't fit in registers when we're passing or any other things like that Uh, and we put push the IPs on there But that's not where the IPs to return to actually come from they come from a thing called the shadow stack right which is like another thing That's kept as like a separate right you know, I'm saying anyone and there's also

1:26:07 There's also they called a return address, but there's all there's a bunch of things in the CPU They're tracking this information. It's not actually from the stack anymore like that's kind of like old news, right Uh, low-level TV saying the shadow stack is rarely used they go in the real stack But they're but it's there. So what happens if that is what I'm asking is what happens if that is on We should just get Ed in here and he should just explain it okay Next episode next episode that's true. Yeah, I want to know what happens if you I want because I don't know like I assume the CPU Actually has like some kind of a path for this or something meaning like it would look and go like Oh someone did modify the stack so I can't just assume

1:26:49 That it's yeah, I don't know like Yeah, like I don't know what they did there and maybe I can probably just go read it like it's probably in the Intel architecture manual right So I will go do that after but I'm just curious if you read it back I did not run into that okay, so it just works. I did the naive thing. Yeah, I did the naive thing that Assumes it is on the stack and then it worked. Yeah, I only know about the shadow DOM so shadow DOM Well, because there's stuff like that where like if you do code pages right if you do self modifying code for example That doesn't really work anymore unless you invalidate the instruction cache right like that was the thing that actually broke because of because of having like

1:27:29 So I didn't know if there would be something like that with all these additional ways that return addresses get checked Because they just assume that no one's patching the return address and so I didn't know if that was an actual assumption And so you would be like oh, yeah, we had to call the like Invalidate the return address stack thing or whatever right and yeah, or if it just works and I guess the answer is it just works My second question was would a data breakpoint work for that you gotta expand I want a data breakpoint if you said a breakpoint if you set a breakpoint where you Said this at this location on the stack. I am setting a breakpoint if anyone reads it or something like this Yeah, I guess you could I guess the reason why you wouldn't want to is is to keep those slots was like open for user breakpoints

1:28:16 Yeah, so so I didn't think of doing that, but I think you probably could do that. I don't know if you would catch the like Will that catch instructions that aren't literal reads but just mechanisms at the point you use is to pull the return address I'm not 100% sure, but it's good idea like with the like it would wrap maybe bypass as the data breakpoint possibly right Yeah, I'm not 100% sure. I don't know. I know nothing, but I'd like to give my I'd like to shoot my shot as well Could you push a like a fake stack frame on that has a single instruction function that is just an int three or whatever the break is And then continue on and let it go and then when it comes back it So it had to bypass through that function no matter what

1:28:58 In one sense like so we actually started this idea back when what back when I was prototyping stepping we started with that idea um We didn't move Just like yeah, we tried that dumb From the nicest way since I'm the guest He literally said they started with it like he's basically saying that primes idea was good and it was the same ideas them and you just like Well, that means primes an idiot All right, I guess he doesn't know anything. I collected elves right now Yeah, it's like you get there from that. He said they started with it like it was there He's you have a great point. You're very smart Shut up TJ just does not want these elves to flow down. He can't he can't handle you having to win prime

1:29:48 He can't handle a w in the in the prime corner. I'm literally wearing an I love TJ Just to like help like bolster his confidence and he's still fucking obvious true um Yeah, so we we did start with that we did push an extra frame onto the stack But we just instead of moving to 911 we just allocated the code page where there wasn't in three and then move and then put it We changed the return address to that But you can save that extra code page injection like it kind of gives you the same effect But when you move to address 911 instead you don't have to inject that extra code page and treat in three is differently Like it kind of just gives you It's just a little bit simpler version of the same idea now you wouldn't I don't think you'd want to push further on to the stack

1:30:36 Because then you're kind of changing like you know how deep is this how deep is their stack are they close to a stack over for are you doing things that are like You know, you're kind of changing the way the program is running a little bit too much. I think in that case um I mean you could do it though you well Yeah, you would probably take that approach prime if you actually had some things you needed to do Like if you had it if you had a really good like like a reason for injecting more code That would probably be the right thing to do right Yeah, so this is you can actually test this too and That's garrow And this is actually how visual studio like if you're in visual studio and you want to evaluate a function

1:31:17 It'll actually do that so it so inside the watch window if you if you say like I want to evaluate this function or In the immediate window probably you want to evaluate a function To evaluate it But it's a function from the program what it'll literally do is take your selected thread And it'll push a new call frame on to it and call whatever function you wanted to run and you can actually test and verify this You can put like uh oh call sleep 1000 or whatever and then if you pause the thread It's in sleep on top of whatever code you were actually actually so it doesn't inject a thread It just uses whatever thread you had injected or you had selected So you may actually does you may not realize this um, but you just insulted me great gravely by saying that I have the same level of ideas as visual studio

1:31:59 Which is I want to be clear. I would never Okay, there's a line between friendly banter and proper insulting and I would never For my good friend Brian I'm so sorry. I did not uh, yeah, I didn't mean to offend. Can we get the primogen subtitle change to visual studio developer He never shows his yeah, even though I write good one for him I know that's because I actually use my real camera as a post because if I didn't I'd actually get the leggy version of Myself talking so I have to use my real camera All right, so Debuggers everyone they're awesome if they're good and they're useless if they're not Right, that's what that's the TLDR for this stream Is like if a debugger is awesome you should use it because it saves you a lot of time

1:32:59 And if it's not it's just a waste of time and you can keep doing your printf debugging because it would be just as good There you go. Well, I actually do there is something inside of me that always comes to the same like kind of conclusion Which is I've also not used enough like Vim daping I haven't dapped hard enough with the boys yet Uh, is that when they do the the thing like this oh Bill Gates says this which is my favorite version of daping but uh when That sometimes I think maybe my speed at a debugger is also skill issues That maybe I need to actually invest more into learning the dap side of things and the navigation being fast with it But I also don't run into that issues very often that it's like

1:33:41 So it's like it's been this kind of a difficult thing, but I need to get pride pride by the way is the uh elixir version of some level of debugging so I'd like to pry it up and and Drop that at print after bugging after this because you guys make debugger seem so like this like exotic place that I got to go But I also don't do game development or C++ already Why don't we do it when you next time you're gonna be doing a project in a compiled language like whenever that whenever that next happens right We should do a stream where we're like we'll do some debugging like we'll show you what we normally do is you can just see like Here's how we do let's do a good seat. Let's let's do like I wouldn't do uh the devil's language C++

1:34:15 Let's do the Lord's language C. Okay. Holden. Yeah. Yeah. Holden Is actually trying to make stuff good for rad debugging so let's do that. Yeah both of those will work I will say though the thing If you're not in like compiled language land like this one of your language has a REPL You can get some of the experience from truth like when you're running elixir and you run it like you can run stuff from the REPL You can send messages you can go inspect data you can do so there is like You know there are some other options to that you can explore even in in some of these other languages Buster and some VM languages have full debuggers, right? I mean You can do full debugging in Java. I think Python has pub G or whatever. It's called that's also like

1:35:04 Cube G is the name of the debugger in my head. I call it pub G But I actually can't remember what it is, but it's like key It's pronounced pub G. It's a video game. Yeah It's completely unless you're playing something completely different than everyone else has been playing pro Crazy All I know is when I saw it I had such severe dyslexia that all I saw was pub G And so now I can't unsee it and so I'm just like oh it's pub G. There's that's what the Python way is that's all I know I for some reason I played player unknowns battlegrounds for many hours and somehow never thought pub G And now I will Be sure if you were on the standup when you were playing it at the same time frame it would be top of mind

1:35:53 You know, we should actually do because I've never played pub G We should do like a Casey introduces us to pub G Call oh god, I mean I have been so long since I played it I would be the worst we just all get shot immediately would be what would happen I'd be like well, I didn't get a chance to show you anything, but sorry guys But we can try like that sure I try next stand up. Let's just play video games for just playing That's actually much closer to a you know, I fang level startup anyways, right? I stand up time for some team building. Let's pick a video game All right team building stream would be so funny Oh, we'll talk about that offline. I gotta go. I got another podcast. Oh, sorry our bad podcast T.J

1:36:35 All right, they did have some good ideas. I wanted to be clear. Yeah. Hey, I appreciate that. Hey Ryan. Where can everybody find you at? Uh, you can uh, I'm on uh, Twitter uh at or X I'm linking it right now Uh, it's Ryan J flurry FL EU RY um, and then I've got a sub stack uh, that I I have a couple of blog posts on on like debuggers and how they work and then I've got a bunch of other stuff uh, that's our flurry FL EU RY.com Um, and then I think that's basically everything Where can they find the rad debugger?

1:37:13 Oh, yeah, uh, you can go to github.com slot. It's it's kind of a complicated URL But it's github.com slot actually can I put it in the chat? Yeah, yeah, yeah, I literally you can also just search for rad space debugger or epic space rad debugger I usually in google it will come up. Yeah, yeah, I guess I don't have oh studio chat. Here we go. I just see Yeah, if someone already linked it's rad games extension rad debugger your fans are in the chat Yeah, you got a lot of hands. Is it a return address from your sub stack 9-1-1?

1:37:45 Uh, no, but it maybe should be yeah, that's a good idea prime You didn't I got one Take that universe all right everybody. Thank you very much for joining us casey computer enhance.com If you want to actually learn how computers work tj He streams by the way, and the name is the prime gen. Thank you for joining us Thank you very much. That was a great outro. I know Thank you. Thank you. Thank you