Software Sessions

About Software Sessions

Practical conversations about software development.

Publisher

Jeremy Jung

Category

education · technology

Language

en

Explicit

No

First episode

13 Sept 2019

Latest episode

6 Oct 2026

Latest episodes

1 of 66 episodes have a transcript.

1. 

   6 Oct 2026

   Theo Steiner on Frontend Development at LINE

   Theo is a Web Developer at LINE Yahoo Corporation in Tokyo with a master's degree in contemporary Japanese literature. We discuss how mapping events in literature led him to software, pitching Svelte to his team, frontend development at LINE, and hosting the UIT Inside podcast. This episode was recorded at the LY Corporation offices in Tokyo. I hope you enjoy it! Theo Steiner Theo homepage (https://theosteiner.de/) 1Q84 (https://en.wikipedia.org/wiki/1Q84) (Murakami novel that got Theo interested in contemporary japanese literature) Where the Magic Happens (https://literary.tokyo/) (Mapping locations from the 1Q84 novel) svebcomponents (https://github.com/svebcomponents/svebcomponents) (Theo's svelte web components library) Sophia University (https://www.sophia.ac.jp/eng/) (Theo's Graduate school) Simultaneous recruiting of new graduates (https://en.wikipedia.org/wiki/Simultaneous_recruiting_of_new_graduates) (The process Theo skipped) What exactly is the Working Holiday programme? (https://www.auswaertiges-amt.de/en/visa-service/buergerservice/faq/06-workingholiday-606672) UIT Inside UIT Inside (https://uit-inside.linecorp.com/) (LINE frontend dev Japanese podcast hosted by Theo) Walking up the ATProto Stack (https://uit-inside.linecorp.com/episode/192) (English episode with Dan Abramov) LINE LINE Restaurant Plus (https://www.lycbiz.com/jp/column/line-official-account/restaurant/service-information/line-restaurant-plus/) (Japanese) LINE Mini app (https://www.lycbiz.com/jp/service/line-mini-app/) (Japanese) LINE Front-end Framework (https://developers.line.biz/en/docs/liff/overview/) LINE Yahoo Corporation (https://www.lycorp.co.jp/en/) Frontend Svelte (https://svelte.dev/) Svelte Japan Offline Meetups (https://svelte-jp.connpass.com/) Japanese Svelte Companies (https://github.com/svelte-jp/japanese-svelte-companies) SvelteKit routing (https://svelte.dev/docs/kit/routing) (file based routing) Introducing runes (https://svelte.dev/blog/runes) (to Svelte) Vue.js (https://vuejs.org/) Vue Router (https://router.vuejs.org/file-based-routing/) (file based routing) Vue Fes (https://vuefes.jp) (Japanese Vue conference) Lit (https://lit.dev/) (Helps build Web components) SolidJS (https://www.solidjs.com/) React (https://react.dev/) Transcript You can help correct transcripts on GitHub (https://github.com/jeremyjung/softwaresessions/tree/master/content/episodes). Introductions [00:00:00] Jeremy: Today I'm talking to Theo Steiner. [00:00:02] We're here at the beautiful LINE offices here in, in Tokyo, Japan. And, uh, I wanted to talk to to Theo because I think he has such an interesting background. Um, a lot of people, they come from backgrounds that are purely just, "I liked computers, so I got into computer science," but I think, your background is a little bit different [00:00:25] maybe you could start with first off how you got here to, to Japan from, all the way from, from Germany, was it? Path to Japan [00:00:33] Theo: Yeah, I'm from Germany. Yeah, that too is a bit of like a unconventional story, I'd say. So many Germans, like when they turn 18, they go somewhere. Like we used to have like military service, but that got, canceled when I... Just before I turned 18. So like a lot of people used that, like newly gained freedom and like they went somewhere like to New Zealand, Australia, Canada, I think are like the typical places to go. [00:01:01] And I actually, I wanted to go to Russia, but back then, it was, it just happened to be at the time where Russia started the war in Ukraine. So my mom was like, "Okay, you can go anywhere, but please don't go to Russia." So I, I checked like which country does Germany have working holiday agreements with. [00:01:23] And the usual suspects came up, and then also Korea and Japan. And I was like, "Oh, that sounds interesting. I'll just text the embassies and whoever writes back first, like I go there." And the Japanese Embassy was like a day quicker than the Korean Embassy. So that's like the whole background to why I ended up in Japan, and I knew nothing about Japan at the time. [00:01:48] So I loved sushi. I always was like a, a huge fish eater. So I knew sushi and I was like, a country with like food that delicious can't be bad, so why not just go? And that's how I ended up here. Like I came for a year and fell in love with the country, made great friends, and yeah, [00:02:10] Jeremy: So it's almost like a, a coin flip. You were saying, "Well, Korea or Japan, whoever gets back to me first." That's, that's quite a leap. [00:02:18] Theo: Yeah. And actually, I think I was like, I knew Korea more than Japan because I, I was like, growing up I used to play StarCraft, [00:02:26] And Korea, all the, like, legendary players were from Korea. So I, I knew a bit about Korea. I knew almost nothing about Japan. So I think, yeah, it was, like, purely luck that I ended up here. Working at farms and snow camp [00:02:39] Jeremy: And how was your, your first year here? what were you doing that, that first year? [00:02:43] Theo: So I came, as I said, on that working holiday visa, and, you're allowed to work almost any job except, in nightlife for obvious reasons. And, I started, working on farms, so I wouldn't receive any money, but I'd receive, food and, a place to sleep. So I worked my way north starting from Tokyo, and, I had this, weird plan of, like, in the winter I wanna be in Hokkaido because I knew they have, like, great snow, and I like snowboarding, so I need to get there by winter. [00:03:14] So I worked myself up, slowly going north from Tokyo, working on, like, a farm that, I think they were, like, growing eggplants and tomatoes. [00:03:23] And then I went further to the north and did, like, a snow camp for kids. [00:03:28] And, and Further to the north, and then I ended up in Hokkaido, and I,I was actually hired to shovel snow. [00:03:36] Which was perfect because it was, like, in a ski resort, and I would get to, snowboard after work [00:03:41] Jeremy: Oh, that's awesome. So it's, it's really interesting because these are almost the furthest things possible from computer programming. (laughs) [00:03:50] Theo: Mm-hmm [00:03:51] Jeremy: So, so how did you go from, uh, running the snow camps and working at ski resorts and farming to, to programming? Majoring in Japanese literature to get back to Japan [00:04:01] Theo: Yeah. So I actually got even further from computer programming doing that. So I, I wanted to stay in Japan, but, my parents were like, "No, you gotta go back to Germany and, do a proper university degree first." And I was like, "Okay, which university degree is gonna guarantee me that I can come back here and, like, reunite with my friends?" [00:04:23] So I enrolled for, uh, Japanese studies with, a major in literature. So I, I actually ended up being a literature major at university. But I did pick... I had this, like... I've always had an interest in computers, like growing up playing StarCraft and stuff like that. I never really knew how to code. Maybe I did some, bare minimum, like HTML editing when I was younger. [00:04:47] But I thought, "Yeah, I need something that can, like, carry, carry my interest in Japan and carry my addiction to sushi when I get older." So I... Some reasonable part inside of me was like, "Okay, let's pick a minor that can be turned into money," and I picked computer science as a minor actually. [00:05:05] Jeremy: I, I take it there aren't too many opportunities for Japanese literature jobs here [00:05:10] Theo: Ah, like some of my friends who studied with me actually pulled it off. Like many of them are like translators. Yeah, but like you have to be like very skillful to be, to like translate literature. [00:05:22] And there's not as much demand, sadly. Literary Cartographic Analysis [00:05:26] Jeremy: And were you, you studying particular classics in literature or what was the, the major like? [00:05:33] Theo: I was actually, um, most interested in contemporary literature and also contemporary literature set in Tokyo. So I was always interested in, like, the, the urban space. [00:05:43] And during the end of my bachelor's degree, I started working towards that. And I thought I wanted to continue to work in that actually. [00:05:52] So I, I picked for my master's degree, I picked a university in Tokyo that would allow me to, like, study in the actual urban environment in Tokyo and, like... I did this thing called, um, literary cartographic analysis, where you basically, you map out a book onto a map, and then you look at the map at the cartography of the story, and you check what kind of, characters appear at which places and, like, how does the urban space relate to the, content and, like, the analytical layer of the work you're working with. [00:06:31] And that's actually one of the points where I got back into touch with my, like, computer science part because, like, you obviously don't wanna do that sort of analysis on a paper map. You wanna map it onto Google Maps or, OpenStreetMaps and, have all those, digital ways of working with your data to, like, make your life a bit easier. [00:06:51] So for my master's degree, I actually, I built... I think it's still up. it's literary.tokyo, and, uh, it's basically this service. It's a system as a service for mapping literature onto Tokyo, which is very specific, but I did it for my master's thesis. And if you ever wanna check it out and maybe do your own research, uh, feel free to go there. [00:07:17] Jeremy: Is it where you were reading modern literature and, and manually saying what locations are people talking about or what hints can I get so that I know where these characters are and where these events are taking [00:07:30] Theo: Exactly. So I was, my thesis was concerning itself with, magic realism. And like magic realism, to the people who don't know maybe listening, it's like this literary genre where like out of the blue something like really weird and magical happens. It's like often associated with, Latin America. But like many contemporary Japanese art, authors are also like associated with this, genre. [00:07:57] Like for example, Murakami Haruki, who is like probably the most famous Japanese writer of all times. He can be like put into this genre. So I wanted to l- see how like the magic realism interf- interferes with the Japanese like urban space and like study if there's like particular spaces where like magic stuff happens. [00:08:24] Like I had this theory that like magic events would happen at like liminal spaces, like perhaps where the like city interferes with the ocean or like where like daylight and nightlife interfere and stuff like that. So I, I tried to map that out onto a map and I actually, I don't quite re- remember the results of my analysis, but yeah, I think I had a lot of fun playing with like maps and literature [00:08:51] Jeremy: Yeah, that's really interesting. I think that, the visualization of I read a book and where did these people go? And like you said, where did these more magical things happen? And yeah, that would be interesting to know if there was a trend of these parts of Tokyo or these neighborhoods for some reason inspire people to, to see those things. [00:09:16] Yeah. [00:09:16] Theo: Yeah, and then it sort of allowed me to connect my passions of, Japan and literature, and then also I've had always had a passion for,computers. So it was like the perfect intersection of everything, and it allowed me to, like, really dive into stuff I didn't know about back then. So I, I had no idea about, like, how databases work, but, building a SaaS for, like, research forced me to, like, learn about SQL and stuff like that, and it, it made a lot of things click and was really a nice experience. [00:09:50] Jeremy: Yeah, I think it's not until you build a project and then you really see how each piece comes together, whether it's, like you said, the database or the front end or back end. And basically everything before is all abstract, and then when it comes time to build something, that's when you realize, "Oh, these are, this is what I actually need to know." [00:10:13] Yeah was there a specific, book or a specific thing from, I suppose, contemporary literature that got you interested in all this? Got interested due to Murakami's 1Q84 [00:10:23] Theo: Yeah. So I think it's a bit cliche, but like it's one of the, the Murakami books. It's, 1Q84. So ichi Q hachi yon is the Japanese title, and it's like a... I'd say it's like a, a romance set in Tokyo, and it really plays with like the, the spatial, arrangements of the city. actually mapping this out on a map was so interesting because, like, there's like two, protagonists, as you would have it with a romance, and like there's three books. [00:10:57] It's a, it's a huge epic of like three books. And like during the first book, like their spheres of action, I think is the term I used back then, didn't like overlap at all because they were like... I don't wanna spoil the book for anyone, but like TLDR, it's like they used to be in the same class in school and had like this,I don't know, like meteorite, connection to each other. [00:11:23] Like they immediately, felt connected, and then they got separated, and then they-- The book is about how they find back to each other in this like world where like a lot of magic events happen. And like books one and two, their like spheres of action on the map don't interlap at-- overlap at all. So like it's almost as if, like they're like separated physically from each other, and then the book and the events in the books like force them to like come closer and closer together, and that's like sort of on a geographic or cartographic level, like how they're like separated and then have to find back together, which I thought was like a, a beautiful metaphor. [00:12:10] And maybe that's intrinsic to like every romance, but it was like really interesting finding that like connection on the map [00:12:19] Jeremy: It's really cool how that also connects to your, your master's, thesis, was it? Where it's, you got interested because of the spatial component and then you you built, software basically researching that, that spatial component So we've kind of gone into how you got to Japan, how you got into programming and your experience with literature. [00:12:45] Where did you go from there? How did you end up here? we're at the LINE offices today. How did you make that connection? Getting into programming and svelte [00:12:52] Theo: So I basically building that, research project, I realized, wait, this is a lot more fun than I thought. It's like building web apps is super interesting. So I I always kinda knew that, being in academia, at least in academia when it comes to Japanese literature, is, like, not the most sustainable passion. [00:13:18] So I wanted to, get a foothold in somewhere, in some area that, was related with, a bit more earning. I started freelancing from there, and, like, I actually did remote work from Japan for, like, um, companies in Germany where, like, friends had connected me with. And during that, I honestly, I had so little technical knowledge, and I wanted to get more knowledgeable. [00:13:44] So I, I started researching online and I remember, like, bumping into React, and at the time I thought React was, like, the most complicated rocket science ever. And I couldn't grasp it. I saw, like, the, the hooks and, like, useEffects, and I was like, "No, I don't, I don't have the mathematical knowledge to do React." [00:14:05] And I, I researched further, and I'm not sure if you're familiar with, Svelte and Rich Harris, but I saw this, talk by Rich Harris that is called Rethinking Reactivity, which is, I think, the talk he did just before announcing Svelte 3. And, it basically, suggests that, like, HTML is the mother language of the web, and therefore a web app should be authored in, like, a subset or, like, a superset of HTML rather than JSX with... [00:14:37] J- JSX, which is a superset of, JavaScript. And to me, as a, like, person who was at the time not comfortable with, like, a lot of JavaScript code, that felt like such a revelation, and I was like, "Oh, wait, I can actually grok this," because this is just HTML, and then you have,the declarative logic sprinkled onto it. Starting Svelte offline meetups [00:14:58] Theo: And I got really excited about Svelte, and I searched if there were, like, any meetups. And there were no meetups in Japan. There was, like, this one, fellow from Canada, Takoyaki, I think, was his name, and he was running this Discord server, Svelte Japan, and I was, I think, one of the first members to join that server, and I was like, "Let's do a offline meetup." [00:15:19] And I think he was based not in Tokyo, so it, it was difficult to, do a meetup with him. But I found a couple of other folks, and I think I organized the first ever Svelte meetup in Japan. It was, like, in the stinky cellar of a Shinjuku hotel. It was pretty interesting looking back. But, uh, yeah, that allowed me to connect with, like- a lot of, web developers in Japan and like with the community, the growing community around Svelte. [00:15:54] And I, at that s- time, I also got more active on Discord, on the Svelte Discord. First, like I came there to ask questions because I had no clue about programming, and I, I asked, like looking back, I asked the most basic questions and like there were so many kind people helping me and like not just giving me the solution, but like help me arrive at the solution on my own terms, and they were like really patient with helping me learn. [00:16:21] And I really appreciated that, and I thought to myself, "I wanna be that person. I wanna be the, the person like helping others understand this." And through that, I got like more and more active on the Svelte Discord, and also more and more on the Japanese community side of things. And I think around that time, I did my first open source contribution to Svelte. [00:16:44] Looking back at that PR, I think it... Like nowadays I would be in horror, like just seeing it. I think I, I committed the like entire node modules to the, [00:16:53] Jeremy: Oh, [00:16:54] Theo: repo or something. But like once again, the Svelte maintainers were super nice about me, and they told me, "Hey, you don't do that." And, uh, that's how I learned a lot. Only applying to LINE [00:17:05] Theo: And yeah, so I guess that was my transition into actual software development, like the, the Svelte ecosystem. And I started contributing and, like, uh, being more a-active. And when the time came and I was about to graduate university, I was like, "I wonder if I can work in this field." And I did some research, and it just happened to be that, the LINE offices were right next to my university. [00:17:35] So I went to Sophia University, which is quite close from where we are, and the, the LINE office used to be in a different building back then, but it's still quite close to where we are right now. And I saw that, and I... It f- felt like fate. no way, it's, right next to my university. So I, I, I thought, "I'll just give it a shot." [00:17:53] I didn't think I ha- I'd have any chance to, like, work at a big tech company, so I just applied there. And, like most, people in Japan, when graduating university, they apply to... They do this, it's called shukatsu, which means, basically job hunting. And, you buy, like, a suit, and then you apply at, I don't know, 30 or 40 companies, and then you go, to the interviews, and it's, a really gruesome process, and I didn't wanna go do that. [00:18:19] So I was like, "It's gonna be LINE," because I really e-enjoyed their app, or no one else. So I only applied to them. And I thought my backup plan was, if it d-doesn't work out, I'll, A, either stay in academia or, B, I'll continue freelancing. And somehow mirac-miraculously it, worked out, and I've been here ever since. [00:18:45] I, I think I've grown a lot as an engineer. I have, amazing peers, amazing colleagues, amazing superiors. I really learned a lot. [00:18:55] Jeremy: that's, that's pretty amazing to, apply to one place and get it. [00:18:59] Theo: It was like a, a one shot. The progression of the Svelte Japan meetups [00:19:01] Jeremy: yeah. yeah.Yeah, so you were running this, this Svelte meetup here, right? And you said it started just in some b- basement hotel. Um, is it, uh... How has it grown? Do you still run in-person meetups or [00:19:19] Theo: So I am still involved with the community, especially the, the Svelte Japan community, but like turns out there are people way more capable of like making connections and like running events than me. And, um, there's this, person called Yuki, and he has taken over since, and he like... We literally changed in entire different levels. [00:19:43] Like we're like now being hosted at like the DeNA offices in Shibuya on like the 35th floor of like some skyscraper. It's like it doesn't compare at all. And so I still, I, I really enjoy going to these events and I, I still enjoy hanging out with all those folks. Yeah. But not as involved anymore [00:20:05] Jeremy: So as an attendee. As an attendee. Uh, yeah, yeah, yeah. Yeah. Less, less work that way too. [00:20:11] Theo: That's less work, yeah What's LINE? [00:20:13] Jeremy: I think a lot of people, if they don't live in Japan, they don't necessarily know what LINE is. Could you maybe talk about what it is? [00:20:22] Theo: Okay, so yeah,that's an important thing to cover. So LINE is basically WhatsApp in Japan, but it's, it does more than WhatsApp does. So probably people listening have heard of WeChat, the like Chinese super app where you can buy stuff and there's like payments integrated into the app and like mini apps running on the app, and LINE is sort of the Japanese, equivalent of that. [00:20:51] I first got in touch with it actually when I came here in 2014, the f- very first time, uh, f- for my working holiday. And back then in Germany, like WhatsApp was, had already kicked off, but like people were only using it to message. And if you wanted to call, you would still use the normal like telephone LINEs to call people. [00:21:13] But I came here and no one was like ever exchanging phone numbers or anything. You would just exchange your LINE, and back then you would exchanging by shaking your phone. And if you would shake your phone in the vicinity of somebody else, the phones would interpret the shaking as you wanting to exchange contacts, and then the people shaking their phones would like pop up in your LINE app and you could add them as friends. [00:21:37] Jeremy: Okay. (laughs) [00:21:38] Theo: It was so cool, and I was like blown away by it. And then also, it has since been retired. I actually don't know why. Maybe it was too goofy. But yeah, I... Like the first time I came here, I would just go, let's say, to the beach with my friends, and there was like a friend I didn't know, and then we were like, "Let's, exchange contacts," and we would [00:21:59] just [00:21:59] Jeremy: shake. [00:22:00] Theo: Each other's phones. Yeah. [00:22:03] Jeremy: Yeah, it, it is interesting how each country has a different, way of doing text messages or or voice. Like you said, a lot of other countries use WhatsApp,um, and then WeChat in China. And I, I think in the United States it feels like people are m- mostly using just iMessage or regular text messages. [00:22:25] Uh, so I'm not really sure how all these things happened, but- Yeah ... each country has picked their own way of doing things, I suppose. [00:22:35] Theo: It's like, it's almost as if they all developed at the same time, and as such, the apps, fit different ecological, niches almost, like ecological... Different parts in the ecosystem depending on what the country needed at the time. And actually, LINE started out just after the 2011 earthquake. [00:22:58] Back in the day, LINE Corporation still was... So it's no longer LINE Corporation, it's LY Corporation nowadays, since we merged with Yahoo Japan. but back in the days, it used to be NHN, which is, the, the Japanese branch of Naver, which is a Korean company. And then, the big earthquake hap- happened in 2011, and, like, people really had, like, issues reaching out to their loved ones. [00:23:28] And, LINE actually got initiated as a project to help connect people after, like, disasters, because the phone LINEs got, massively overloaded after the disaster. And, like, it was thought that if you used, mobile internet, you could, fit a lot more bandwidth into the same slot. So that's, uh, the, I think, initial thought that started LINE app, which is a very nice thing, yeah. [00:23:57] And it just exploded from there. nowadays, I think you would really have to look for a while to find a Japanese person who doesn't have LINE installed on their phone. It's just, the default messaging app. [00:24:10] Jeremy: Yeah, it's interesting that you mentioned that it was the, the Japanese branch of Naver, which you said was a Korean company, 'cause I think Korea, they also have their own messaging systems that they use. Is, is there one by Naver as well? [00:24:25] Theo: There's, it's not by Naver, it's by Kakao. [00:24:27] Jeremy: Kakao. Oh, okay okay. [00:24:28] Theo: the Korean one. [00:24:29] Jeremy: Okay. KakaoTalk, right? yeah, yeah. [00:24:32] yeah, everybody's got their own, so now we have to have all the different apps on our phones. [00:24:37] Theo: I have like, yeah, I think I have at least 15 messenger apps on my phone, including Signal, Line integrated browser for businesses [00:24:44] Jeremy: That's right. Yeah, those have gotten big too. Yeah. at least for your part now that we've kind of explained what LINE is, um, what do you work on at LINE? [00:24:55] Theo: So there's this thing called official accounts, which means basically if you run a business, you can have an account on LINE, and you can have people add you as a friend, and then you can, like, do customer relations via that. You can, contact your friends and send out coupons, send out,reward cards and stuff like that. [00:25:16] And, those are deeply integrated into the native app, but a lot of the, like, additional features run as mini apps on the native app. And the native app has, a special browser. It's called the Liff Browser. If you're ever curious, you can Google it. There's, an, a API that has been published and everything. [00:25:34] It's L-I-F-F, and, it's short for LINE Integrated Frontend Framework. And it's basically this special browser where once you open it, your auth token is automatically injected. Mini apps don't need you to log in. They can just reuse your LINE login for, doing stuff. And I work on a team that maintains, quite a few of those mini apps, and we're only six people at the time, but we... [00:26:06] I think we maintain, seven or eight quite big mini apps, which is interesting because, like, with LINE, since basically everyone in Japan uses it, everything has, like, huge scale. You roll out something new, and you're like, you check, like, Sentry or something for the logs to see how everything is doing. [00:26:26] And, instead of, having a few hundred requests, you get, a few hundred thousand requests or something. It's, it's a really, interesting scale of things, and it's really rewarding because when I go to, like, a restaurant or something, most places actually use the apps that I build for work. [00:26:44] So it's really easy if I go with friends and they ask, "What do you even do?" I can just point at something and like, "This screen, I built this screen," or, "I made this button green," or something. [00:26:55] Jeremy: Can you give an example for people who haven't seen these apps, like a specific way that they would see it and what it did? Order with LINE [00:27:04] Theo: So I think the most interesting app is the one I've been working on recently. It's it's called Order with LINE, and it l- allows you in restaurants... They would put up, either a, a QR code or like a NFC thing that you can touch to start the app, and then it opens, this mini browser inside the LINE app, and you can order straight from your table. [00:27:26] In Japan, it's, surprisingly common now to, like, not talk to your waiters anymore, but instead order from, like, a little mini app. And, it's, integrated with the a- with the app in that, for example, if you forgot to send your card, you will get, a notification, a native notification from your app that you forgot to send your card. [00:27:46] Or, you can look at your... even after you've paid, you can look at the things you ordered because it still remembers you via your LINE account [00:27:56] Jeremy: Okay. Yeah. What I've personally seen a lot of times is it's a website outside of an app, but you're saying that, some of these businesses, when you scan the QR code they give you, it actually opens up LINE and has you do those transactions inside of LINE [00:28:12] Theo: We actually launched two months ago or three months ago, so it's still rolling out, but I suspect the next time you're gonna be in Japan, you're gonna see it a lot more often. I hope, I hope. [00:28:23] Jeremy: yeah, it's always a little tricky too where sometimes depending on where the business is, maybe your connection isn't so good and you're trying to do the QR code thing and it's like, "Hmm." [00:28:36] Theo: Yeah. I try to be as, uh, thoughtful with, people's data as possible. So I, I actually... I got to be tech lead on this project, so of course I picked Svelte, which is, famous for smaller bundles than React. So I thought, if people... once a store adopts this app, people are basically forced to use it, right? [00:28:59] Because it's the default way of ordering inside the store. And whenever you have those sorts of situations where people can't really pick which app they wanna use, I feel like you have to be really responsible about, the amount of JavaScript they have to download, the, like, amount of, like, annoyances they f- face, when using your app. [00:29:21] So we made it explicitly so you don't have to use it within LINE. For example, if you're a tourist or something and you don't have LINE, you can, uh, open it in a web browser. And then also, like, you don't have to add the store as a friend, because, some people think that's annoying or they don't wanna give away their, contact to a place. [00:29:40] So I was very conscious about that building this app [00:29:44] Jeremy: Yeah, that makes sense. I actually remember, uh, this trip, there was a conveyor belt sushi place that to see what place you were in line or, or, yeah, in a literal line in the queue, uh, yeah, they wanted you to add them as a friend, um, so that they would message you your position in queue. And,I mean, I did it, but I also felt like that's a little odd, not, I'm not your friend, right? Uh, in terms of the business anyways. [00:30:14] Theo: I mean, it makes sense from, like, a small business perspective. They, they want you, to become a recurring customer, and then they, if they have your contact, they probably won't even do anything nefarious with it. They just wanna, like, remind you of their existence [00:30:29] Jeremy: But that's good though that you, you've probably personally had your own issues or annoyances where you have friends come visit who, who don't have LINE or aren't familiar with it. So letting them be able to check out and not have to go make an account and all that, I think that's that's really helpful, especially for people who don't necessarily live here. Getting the team to use Svelte [00:30:52] Jeremy: So you said you used, uh, Svelte for this. was your team all familiar with Svelte before you started this project? [00:31:00] Theo: They were not. And I had to do a lot of convincing work. actually I, like ever since a previous project, I wanted to use Svelte, and I didn't just wanna use the framework Svelte, I wanted to use the meta framework that is, available. It's called SvelteKit. It's basically like what Next.js is to React. [00:31:19] And, being young and foolish, I wanted to like go all in on it, like because it was the hot new stuff and I wanted to try it. But my manager actually, being the voice of reason, he was like, "Wait, so you are familiar with this. You have contributed to Svelte. You're like a Svelte stan. You read all the news and everything, but the other folks on this team are not." [00:31:41] Like, we usually work in Vue, which is a great framework as well, I think. But like,I, I just wanted to work with Svelte, and I think there were like a couple of good reasons why we should have picked Svelte, because like the bundle size at the time was even smaller, and then it featured like a lot of, features that made the end user experience better. [00:32:04] So it's, really easy to build,nice UIs with, like, animations and stuff in Svelte, and then it w- had, like, really sophisticated bundle splitting to make, the bundle sizes when, when downloading the app, as small as possible. And,it also had, like, great developer experience, so I wanted to get my colleagues to experience that as well. [00:32:22] But I think my manager was right at the time to, like, sort of stop me and be like, "Hey, this is not the right time to, just, introduce something new." So we instead did something different. Starting with file-based routing [00:32:35] Theo: So SvelteKit, features file-based routing, and at the time my team had never, worked with, apps that had file-based routing. [00:32:45] And, recently even, Next.js and, most meta frameworks have switched to this model of, like, sort of declaratively, stating which routes you have based on your file system. So my, my boss was like, "Why don't we just, try to get this one part that SvelteKit offers and, get, everyone familiar with that first, and then evaluate if we wanna take the next step and, go on and, maybe use SvelteKit in a future project." [00:33:12] And I think, like, looking back, I think that was what... why I was junior and he was senior. Like, I really grew a lot, due to that experience. But, like, we, we picked this library that allowed us to have file-based routing in Vue, which, at the time it was called unplugin-vue-router, but, now it's called... [00:33:32] it's just Vue Router, like Vue Router has merged with them or with it. And, yeah, the team, it was really a slam dunk because the team really enjoyed the way of how it, how it made routing in the app obvious, and how you no longer needed to look at, like, a huge imperative file to understand which routes are, like, routed to in which situations. [00:34:00] And it also introduced this concept of loaders, which a lot of frameworks have nowadays, where, each route declares, like, a, a loader that runs before the router, enters the page, and it made, working with data a lot easier in, like, SPAs. So yeah. it took some convincing and, we did this, first step of, I don't know, this was probably three years ago, where we introduced, file-based routing at first, and then the team really enjoyed it. [00:34:30] And w- then we also started with, web components, and, at the time I felt like Svelte had the most compelling front-end framework, integration with w- web components and, building web components. So we picked Svelte for that as well. And without,diving into Svelte, we got the team familiar with the technologies required for SvelteKit. [00:34:50] And then I think I, I can't really take any credit for that. that's all my managers working, and I learned so much, doing that. But yeah, when we started the current project, the Order with LINE project, a year ago, everyone was familiar with Svelte. Everyone was familiar with file-based routing. [00:35:09] So at that point it was just like a matter of putting the two together and actually adopting SvelteKit, and I think the team has really enjoyed working with it [00:35:18] Jeremy: So the file-based routing that was in a existing project that you introduced it, and then when you were talking about the web components, is it where you were using web components in a project or was it where they were just getting familiar with them? Web components to make components sharable across projects [00:35:33] Theo: No, it was, uh, it was a different project. My team has way too many projects. Uh, if you, if you wanna hi- work with us, we're hiring. yeah. So we have... Having that many projects, we kinda wanna share a lot of the UI code we have, and we were always looking for ways to do that. We initially started out, sharing actual Vue components, but that is fragile because you need to care about, like, are the runtime versions compatible, and stuff like that. [00:36:03] And then, other projects that were not directly affiliated with us also wanted to start using our, like, shared components, and that's where everything breaks down, because you can't just, run Vue inside React or... yeah, it just crashes and burns. So that was the reason why we, we started, like, an internal component library that compiled to web components. [00:36:29] So we didn't wanna author web components because, even if you use a framework like Lit, the DX of that is quite manual and quite involved, and we wanted to be at a higher level so we can move faster. We didn't have that m- many, really complex, like UIs, so it was fine to work at a high level. [00:36:51] And for that, we picked, uh, Svelte simply because we wanted to get familiar with it. And, so this is actually no longer true for Svelte 5, but at the time, Svelte 3 and Svelte 4, they also had by far the lowest, the smallest bundle sizes for web components when w- when compiling to web components. And I thought that was really attractive for the thing we tried to do, so that's why we picked Svelte. Svelte added a runtime and increased bundle size [00:37:16] Jeremy: Hmm. And when you say it's no longer true, does that mean the si- the bundle got bigger or? [00:37:20] Theo: actually, so, uh, Svelte, like Svelte 4 to Svelte 5 was quite the breaking change. I don't know if you're familiar, but they introduced something that's called runes. it's a new way of authoring, reactivity inside the Svelte components, Behind the scenes, it's very similar to how a lot of front-end frameworks have started to do things, which is they use, signals. [00:37:46] So Ryan Carniato, the, the creator of SolidJS, which is like the, the underdog under the frameworks, he, popularized this new paradigm of reactivity where, you're no longer, like, diffing parts of your app, but it's mostly about, really fine-grained reactivity, and you have, a, a signal that tr- directly triggers, like, derived effects or effects. [00:38:17] And, that comes with, a bunch of benefits, and, is actually way faster, so that's why, Svelte changed to that model. But the downside or the trade-off they had to make is now Svelte used to, like, compile away. It would have no runtime, but it, it would... At compile time, it would, like, sort of inline all the reactivity. [00:38:42] But, that doesn't work anymore if you use signals. So now there's, a small... It's a quite the small runtime, but it does have a runtime, and when you ship, web components, you usually have to include that runtime into every web component, so the, the baseline of how small a web component can be as in bundle size becomes bigger. [00:39:04] And yeah, I think that's basically the only trade-off they made with the move to signals, and it kinda kneecapped the web component story, but I still think it was the right move to make for them. [00:39:18] Jeremy: So in your own projects, you would choose to use signals and just take the, the hit on the the size? [00:39:24] Theo: Yeah, I think, I think it depends on what you wanna do. if you really wanna get the smallest bundle size for your users, you should probably author your components by hand. But that that comes with, massive cost in terms of developer experience, because writing web components, even nowadays, still is no treat. [00:39:45] There are so many rough edges, and, if you use something like Lit, it becomes a bit more bearable, but it's still very, low level, and it, it's still very imperative. And if you wanna do, declarative development of, uh, web components, which I think is, a huge gain when it comes to,developer ex-experience and productivity as well, as in how fast you can move, I think picking, like, a framework like Svelte is the right thing to do. [00:40:13] And actually, I recently created a open source project, svebcomponents. The name is terrible, I know, but it's basically this library to make working with web components in Svelte even easier, and, creating web components with Svelte even easier Works on frontend, the backend and scaling is for another team [00:40:30] Jeremy: One of the things you mentioned earlier was that working with the scale you do here, you saw a lot of new problems. Can you give an example of one of those and how you, you addressed it? [00:40:43] Theo: So the, the best thing about my job here is that I only do front-end development. So the people who have to worry about the scale is not my team. It's other people. It's the back-end people, and I think they, they face way more problems with that. For us, it's mostly... Like, for... If you're building an SPA, you don't really care if it's, 1,000 users or a million users that, use it. [00:41:09] I mean, you wanna be careful about, not being wasteful with, API calls, and you don't wanna DDoS your own server. So obviously we're conscious about that. But honestly, there's not too much besides from that that we have to be careful about. It's quite comfortable. Yeah. [00:41:28] Jeremy: It's, uh, I imagine the, the backend team would have a very uh, story to tell. [00:41:33] Theo: Definitely. I mean, they're the ones with, like, the pagers ringing at night [00:41:38] Jeremy: Uh, okay. okay. So you don't, you don't have that, I guess. [00:41:41] Theo: I have a pager, but it never rings because, [00:41:43] Jeremy: That's good. That's good. (laughs) [00:41:46] it's hard to even think about just, the amount of people using, just the chat part of the app,it has to be getting used all the time so frequently, so there's there's gotta be very interesting, uh, architecture decisions there. Hosting UiT Inside Podcast [00:42:01] Jeremy: one thing we haven't talked about is you actually have your-- or you host a, a podcast yourself. could you tell us, like, a little bit about how that started and what that's about? [00:42:11] Theo: So that's... Yeah. I host the UiT Inside podcast, which is mostly in Japanese. Sometimes we do, episodes in English. and that basically started at work. So UiT is the name of the front-end organization at LINE. And one of my good friends and colleagues here, Alan Davalos, I don't know if he's gonna listen to this, actually asked me about, wanting to come on for an episode, I think back when Lit version three was first released, and we just chatted about Lit. [00:42:46] And it ended up being really fun, and I had... Yeah, I really enjoyed talking about tech. I'm really passionate about tech, as you can probably tell. so I, I sort of got hooked, and I, I continued making episodes. And at first, I, I came on as a guest, and I was just, like, the sidekick to the main host and would,throw in, things I had picked up on the internet or, like, uh, bring up, my own experiences. [00:43:14] But then I, I moved into becoming a host myself, and you don't believe, like, how nervous I was, like, being a host in Japanese in, not m- not my mother tongue and, like, speaking to really, amazing people in tech. So I've had, I've had on people like sapphi-red from the Vite core team. I had people on like, um, Anthony Fu, Dan Abramov, like, all those people I look up to. [00:43:41] And I think that's also one of the other perks of, hosting a podcast because you can... I think we talked about this earlier. You, you get to, talk to people you otherwise would have had no chance to meet or, like, talk to. It's such a privilege. Yeah. I've also gotten to go to,conferences to, like, gather, content for the podcast. [00:44:00] So I've been to Vue Fes a couple times, which is such an amazing experience. if you're even slightly, like, involved with the Vite, Vue or, general web dev audience, I think you would have a blast at Vue Fes. And then, Google I/O. I went to Google I/O three times and did, like, podcasts about that as, as well. [00:44:22] It was really interesting. And, like, seeing how a, big corporate conference works in the US. Yeah, I I slid into it and, things escalated from there and yeah. English usage at LINE [00:44:35] Jeremy: Yeah, I think something that you mentioned that's interesting is that most of the recordings are in Japanese with a few that are English. For the, for your coworkers here at at LINE, would you say that most of your business is done in Japanese? Japanese Like, what is the... Is English used here? Would they understand the podcast that you recorded, for example, with, Dan Abramov? [00:45:02] Theo: so when it comes to English, some teams at LINE use English actually as their primary language, but that's mostly, like the native teams that work on the actual messenger app. for my team, it's like 100% Japanese, and I always joke that, the people who start at LINE are the brightest people in Japan that don't speak English. [00:45:28] Yeah, a lot of people obviously can read English, and they have no, problems, like reading, like English documentation or something. But when it comes to something like a podcast where it's like native people speaking like full on speed, and there's like also a lot of like, difficult ways to formulate something or I don't know. [00:45:52] Like for example, with me, I'm not native. I'm not a native English speaker, so like sometimes I end up phrasing things in a weird way. And I think since my native language is like a Germanic language, which is the same family as English, it still somehow works. But like people really underestimate how different Japanese is from English. [00:46:12] Like they're so completely different, like nothing is similar. And I think like a lot of people are like quite harsh on Japanese people not being like in the average, not being great English speakers. But man, those languages are so different. And yeah, you, you don't pick that up just from like standard ed-education [00:46:36] Jeremy: Yeah, it's always different learning a language in school. I'm from the United States. We all take foreign languages, but I would guess that the average American's ability to speak one is lower than the average Japanese person's ability to speak English. So, don't be too hard on yourselves, I guess. Yeah. But that is interesting though that it, you, you're saying it's on a team-by-team basis that that some would communicate in Japanese and some would communicate in English [00:47:10] Theo: Maybe 95% of the company, like the daily communication is in Japanese. And then for the rest of the 5%, it's not even all English, it's Chinese and Korean. [00:47:21] And then I think English is this tiny blob where like some teams speak in English, but it's honestly just a few teams. [00:47:28] Like it's a very big company. I just, I don't know like the exact numbers, but I think there's like over 20,000 employees. [00:47:37] Most of them don't speak English on a day-to-day basis. [00:47:42] Jeremy: since you would say 95% speak Japanese,uh, I, I assume that the vast majority of the employees are probably native speakers as well then, yeah. I'm not sure how true it is, but somebody was telling me that, was just in engineering teams, but at Rakuten they were trying to use English more natively, I guess, within the company. [00:48:04] Theo: Yeah, I think Rakuten made the decision to only speak in English. the, the company is English first, and I think Mercari, the, like, eBay of Japan, so to say, also made the same decision. So there's a few companies, PayPay as well, I think, that, I think it's mostly about the ability to hire foreign talent [00:48:25] Jeremy: They decided that English should be their, primary form of communication, yeah [00:48:29] yeah, I guess we've kinda covered a lot. Is there, there anything else you wanted to mention or think we should've talked about? [00:48:37] Theo: Yeah, if we still have some time, I would love to talk about like at Proto or something because [00:48:42] you reached out to me not on Twitter, but on Blue Sky, which I think is kinda uncommon, so [00:48:47] Jeremy: Yeah, that's a whole-- thing that's going on. [00:48:49] Theo: It's a huge rabbit hole, so I don't know if we have time to get [00:48:52] into that [00:48:54] Jeremy: not today, but yeah, yeah, maybe sometime in the future. That would be a fun one. [00:48:58] Theo: Yeah. I'm pretty AtProto pilled, and if I can like shamelessly plug the, the podcast I recorded with Dan Abramov was on like the AtProto, [00:49:06] Jeremy: [00:49:06] Theo: protocol. So if someone is interested, please feel free to listen to that [00:49:12] Jeremy: For sure. Uh, we'll put that one in the show notes. And that one is in English, so don't don't worry. (laughsc) Theo, if people wanna see what you're up to or check out your podcasts, uh, where should they look? [00:49:24] Theo: So I think the easiest place would be my homepage, which is, T-H-E-O-S-T-E-I-N-E-R.de (http://T-H-E-O-S-T-E-I-N-E-R.de), so theosteiner.de (http://theosteiner.de) [00:49:35] Jeremy: Theo thank you so much for for joining me today. This was fun. [00:49:37] Theo: Thank you so much for having me on

2. 

   11 Mar 2026

   Scott Hanselman on AI-assisted programming

   Scott Hanselman is the the VP of Developer Community at Microsoft. We discuss the ambiguity and non-determinism of agentic loops, expressing clear intent to steer models, the importance of knowing fundamentals, and why saying you vibed something means you don't care. This is an extended version of an interview posted on Software Engineering Radio. Related links Scott Hanselman (https://www.hanselman.com) Hanselminutes Podcast (http://www.hanselminutes.com/) Scott and Mark Learn To (http://scottandmarklearn.to/) Github Copilot (https://github.com/features/copilot) Claude Code (https://code.claude.com/docs/en/overview) Gemini CLI (https://geminicli.com) Codex (https://chatgpt.com/features/codex) OpenCode (https://opencode.ai) Everything is a Ralph Loop (https://ghuntley.com/loop/) Open Live Writer (https://openlivewriter.com) LM Studio (https://lmstudio.ai) Foundry Local (https://www.foundrylocal.ai) Ollama (https://ollama.com) Beads (https://github.com/steveyegge/beads) OpenClaw (https://openclaw.ai) Transcript You can help correct transcripts on GitHub (https://github.com/jeremyjung/softwaresessions/tree/master/content/episodes). Intro [00:00:00] Jeremy: and then do you have a, a hard stop time apart [00:00:02] Jeremy: from what we set? [00:00:03] Scott: No, I'm chilling. I'm, uh, I'm gonna, after this I'm gonna go on Facebook Marketplace and buy a Mac mini [00:00:09] Scott: So that's what you're, you're between me and my mac Mini. [00:00:12] Scott: Should be sitting on their porch. [00:00:13] Jeremy: all right. yeah, I guess we'll get [00:00:15] Jeremy: started. [00:00:16] Scott: Alright, do it man. [00:00:18] Jeremy: Today I am talking to Scott Hanselman. He's the Vice President of developer community at Microsoft. [00:00:24] Jeremy: And I last spoke to Scott back in [00:00:26] Jeremy: 2021 where we discussed NET [00:00:29] Scott: Hey man, how are you? [00:00:30] Jeremy: Pretty good. Today we're gonna talk about all these AI assisted development tools that are out there, and I think it's super overwhelming, I think for a lot of our listeners and for myself. [00:00:43] Jeremy: So let's kind of give an overview of maybe the, the types of tools you have experience with. I think maybe we could start with just the chat bots, the, the ChatGPTs and the text box. you know, go back and forth on Syntax highlighting and autocomplete [00:00:59] Scott: I would even go farther back if I may. and like, not to put too fine a point on it, but like, let's go back to like 1975. 'cause I was thinking about this yesterday. I was trying to explain this to my wife and it's like when I started coding in like 80, 84, I went from assembly to C and people would be like. [00:01:17] Scott: C? Come on, that's too high level, man. You gotta get down to the metal right? You're like, you're, you're, it's gonna rot your brain. C's gonna rot your brain. And then in the early nineties, we got color. And by that I mean we got syntax highlighting and they're like, oh, it's gonna rot your brain. And then we got, Stack Overflow and they were like, oh, you're gonna rot your brain. [00:01:34] Scott: And now we got this. When we started thinking about AI augmented coding, it was next token prediction. You type A for loop, you hit space, and then it makes like ghost text and suggests. So that's the first thing. Then there was the chat bot. So just right before the chat bot, there was the ghost text. Uh, hey, you know, uh, here's, you know, we were, it would like interrupt you. [00:01:57] Scott: And, um, they would have like next token prediction within the editor. And some people really like that 'cause it's like, oh, I know what I'm doing and then it's just gonna auto complete. So it was like auto complete on steroids. Then people went into chat, GPT and started asking questions, and then they would copy paste not from Stack Overflow, they'd copy paste from ChatGPT. The ambiguity loop is non deterministic [00:02:17] Scott: Then the agentic loop started. And that's where things got interesting. And if you really want to sound smart in meetings, uh, just say agentic as many times as possible. And agentic loop sounds even smarter. But the idea was that why am I copy pasting this code from ChatGPT when I could just let the agent, the, the LLM run, build, look at the text, see a warning, fix the warning, and then, and then loop. [00:02:43] Scott: And that's when we started getting into things like GitHub Copilot CLI and Gemini and Amp and OpenCode and, and Claude and all of those kind of things where it's LLM sees some tokens, gets the result, makes a judgment, and it's, I call it the ambiguity loop. Because this is what loops, this is what this is The thing about programming versus LLM programming. [00:03:06] Scott: Programming is not ambiguous. Like it runs exactly as you wrote it. And if there's a bug, it's your fault. And if you want a for loop, you write the for loop, you do it in bash or power shall or whatever, and it does the thing. But an ambiguity loop is like, yeah, I don't really know. Like things could happen and we fool ourselves into thinking like regular expressions and stuff, kind of like handle ambiguity. [00:03:28] Scott: But if you imagine this kind of slider bar of ambiguity between I'm parsing this binary and it's gonna work this way, and here's the way that the bites are packed. Versus the like, I don't know, man, it's unstructured data. What are you gonna do? LLMs are really good at ambiguity. Turns out the toil part of programming is really am ambiguous. [00:03:49] Scott: Like I'm writing this, this bot on this other machine over here. I'm running what's called a Ralph Loop that we'll talk about later, and it's bumping into all kinds of little weird problems. And those little weird problems are just crap that I would have to go and Google or look on Stack Overflow. It's just ambiguity. [00:04:04] Scott: So I'm running not a Ralph Loop, but an ambiguity loop that's like working that out because that's not the fun part of programming. The fun part of programming is like building stuff. So I think that these agents and why people are freaking out about them is because it's that same feeling that we got when Stack Overflow happened and we're like, oh, programming's over. [00:04:21] Scott: Or when, when I got syntax highlighting and I was like, oh, it's, I don't have to do anything now that all the text is in color. Does that make sense? That's how I contextualize it across a historical context. [00:04:32] Jeremy: Yeah, I think that's good to, to bring it back to the syntax highlighting and the auto complete, because I think nowadays that's very uncontroversial, right? People, they for the most part will accept like, oh, the syntax highlighting is there. It helps me. [00:04:49] Scott: Yeah. That's a good way to put it. [00:04:50] Jeremy: Point out what I'm looking for. The auto complete. I can quickly see is the suggestion something I want, yes or no? I think for the most part, people are, are on board. I think this ambiguity loop you're talking about is maybe a little bit I think people have more mixed feelings about is, is because of that what you're describing where it's like this slot machine, right? Like maybe it'll give you a good answer, maybe it won't. One shot without specificity doesn't give good results [00:05:16] Scott: Yes. And here's a really great, I had this internal conversation with this guy, like we're on teams and this is an internal guy. And he was so excited because he did a one shot. A one shot is when you type one prompt and then software pops out. [00:05:29] Scott: He did a one shot and he cloned Minecraft. And you know, there's always this, there's this thing that, um, I think it's called, like there's a paper, I think it's like Project Unicorn or something like that, where like OpenAI cloned Minecraft. But forget about that result for a second. And think about the simple semantics of typing a prompt. [00:05:49] Scott: Make me a clone of Minecraft that runs in the browser using ThreeJS. Which word in that sentence is doing the heavy lifting, the heavy semantic lifting. There's Minecraft. And there's ThreeJS because you're steering it and saying, use ThreeJS versus using Babel versus using whatever. But in that sentence, make me a clone means a copy of Minecraft means a lot. [00:06:16] Scott: The word Minecraft semantically is effectively a 50 page spec. So I was trying to tell the guy, dude, that word is doing literally all the lifting, try explaining to it to give you a clone of Minecraft. Except you don't get to use the word Minecraft. Right. And now that would be effectively impossible. Now you go from, oh my gosh, I did a one shot of Minecraft to, this is a hot mess and I really need to be specific. The job of a programmer is to express understandable intent [00:06:45] Scott: And you're gonna get, like you said, a roll of the dice. So we need to ask ourselves, when we are making these ambiguity loops, what is the job of the programmer? Well, it's to be specific. Because it's all about expressing your intent and making your intent understood. And I'm sure we would've loved in the sixties, seventies, and eighties to like just talk to the thing, but it wasn't specific enough. [00:07:11] Scott: So we needed to be so specific then like put that electron in that register and we're gonna use assembly to do it. And now suddenly I can say, Hey, you know, computer, what's my blood sugar? And it knows to go into a rest API to get the tokens to go up to my Mongo database to bring all that kind of stuff down. [00:07:29] Scott: 'cause suddenly the, the Lego pieces are, are, are high enough and the stack is deep enough that I can get what I want. And it might try it in Python one day, it might try it in JavaScript one day, but the result is I get to know what my blood sugar is. 'cause a rest, API is called. But that ambiguity loop shouldn't be something that we should take for granted because it's not gonna work every time. [00:07:53] Scott: And that's why I think it's interesting to use the. The ambiguity loop to get me specific software that runs absolutely reliably because talking to chatGPT or talking to Claude, that's not a scripting language, right. It's effectively a pros compiler, and that's different. So I'm using it to make grounded scripts that I then run rather than just yapping at the thing and pretending that it's bash files. [00:08:19] Jeremy: Yeah, in, in some ways it's like that shift you were talking about earlier where you went from writing assembly or machine code to writing compiled languages compiled down to that. And then we got to higher level languages like JavaScript, and then now we're one level. Even before that, we have our english prose that's helping us compile or generate the, the, high level language. [00:08:46] Scott: You know how people use like Lua or C Sharp and they put it, build it into like Unity apps. Right. And that's like, that's the scripting language for their game. That makes sense to me to embed an LLM would be to embed ambiguity, which they think, oh yeah, look, the NPCs are gonna be able to do stuff. [00:09:04] Scott: But now they could do anything. They could say dumb stuff. They could go off the rails and the game could completely do something that you don't want it to do. That's what we don't want to do. We don't want to push out slop because rolling the dice is not software engineering. You can't one-shot a large app, you need to steer [00:09:18] Jeremy: And I think maybe another important point is with that example where you said somebody tried to one shot, let's, let's write Minecraft. Right? My guess is that the end result was not actually Minecraft. It was probably something that resembled it a little bit, but [00:09:35] Scott: Yeah. I wish I could show you. It's actually, to be clear, it's super impressive. it's a full voxel world. It generates the voxels, it runs in the browser. the textures are all there. [00:09:45] Scott: There's no crafting in mine. Um, but you know, you've got an inventory, it's. It's what you, it's Minecraft if you woke up from a dream and you're just like, oh, I had a dream. I was in Minecraft. So it's maybe like 60% of Minecraft, and that's pretty cool. But again, the spec lives entirely in the word Minecraft because if you said, make me a clone of fufu, and fufu isn't a thing, you're gonna just roll the dice and it's gonna go, it's gonna go insane. [00:10:14] Scott: That's why it's not a reasonable demo to say, make me a space invaders. Make me a Pacman, make me a Donkey Kong. 'cause those words do all the heavy lifting. [00:10:23] Jeremy: And I, I feel like, some of the higher levels people who are running these businesses, CEOs and things like that, they have this vision that. I'm gonna be able to ask this rather ambiguous thing and, and get a complete product, but to me it feels like we're not, we're not actually there. [00:10:41] Scott: Well, so. Like a, an example that I really like to use. Um, and then may I ask, do you have a car and do you, do you [00:10:48] Jeremy: I do, yeah [00:10:50] Scott: Okay. Do you drive stick or do you have auto automatic, automatic. Do you know how to drive stick? [00:10:54] Jeremy: I don't know it well enough to comfortably do it. [00:10:58] Scott: Okay. Do you feel that it is a personal failing, that you don't know how to drive stick. [00:11:01] Jeremy: I do not. [00:11:03] Scott: Okay. So take everything that we just talked about and now apply it to programming, right? Like, I could do assembler, but I couldn't do it without Googling anymore. Do I feel bad about that? No. Could I drive it in an emergency? [00:11:15] Scott: Yes. Could you drive someone to the hospital and stick? Yes. You could probably make it happen if, if you were in a pinch. Here's the question though. What if you answered just now I don't have a car. I only know how to Uber. Then your relationship to the vehicle fundamentally changes. And if the vehicle breaks, you're gonna get outta that car. [00:11:35] Scott: You're gonna leave that poor gentleman there, he's gonna have to fix that car, he's gonna have to repair it, and oil change and tires, and you're just gonna get in another Uber and fly away. But how deep is the stack for you to simply move your body from point A to point B and you have no control over how the guy's gonna get you there. [00:11:52] Scott: So I think that there is value in driving stick occasionally, uh, trying to get somewhere without a GPS, because then the ambiguity has to be, and the responsibility is on you, the onus is on you. While, right now the idea like, Hey, send a stranger in a car with candy to pick me up. and I just like lean back that's the same thing, like asking an Uber to send you somewhere and asking, Claude code or GitHub copilot to make something for you. [00:12:21] Scott: With no specificity. You're lucky. I mean, I hope you get to the airport. Good luck. You know what I mean? [00:12:27] Jeremy: Well, I suppose in that example, the, the Uber or the Lyft, you're giving an address most likely. And it's, it's sort of this well-defined problem of I need to get this person to the airport, or I need to get this person Right. And I, I feel like that's a little different than me saying, telling an LLM, please build me a web browser or build me minecraft. [00:12:53] Jeremy: Minecraft [00:12:54] Scott: that's fair. I think an example would be, uh, Hey, Uber, take me to the airport. [00:12:59] Jeremy: Without specifying which one. [00:13:01] Scott: Without specifying. And there's no ask user question. [00:13:04] Jeremy: So I guess your point maybe is that it's somewhat unrealistic to say something so vague and not expect there to be a back and forth, to be clarifying questions, that kind of thing. [00:13:14] Scott: Exactly. Exactly. Exactly. And I'm sure you've been in an Uber and you've told them, Hey, take the 405 because you are steering them. You're not driving the car, but you steered. And that's why we're finding steering in these CLI coding agents so delightful where it's like I, I, you're 90% in the direction I need you to go. Just get off on this exit. Yeah, those little moments matter specificity matters. Asking questions turned a complicated project into a browser extension [00:13:40] Scott: I'll give you another example. My son is 18 and he has a, a Depop, so he sells like vintage clothing on Depop, which is like a marketplace for people to sell clothing. [00:13:49] Scott: But he ships on a website called Pirate Ship. He doesn't use the beat Depop built in shipping. So I had this vision to make it so he didn't have to copy paste stuff back and forth. And I thought I was gonna do it with like playwright and I was gonna automate browsers and I was gonna build a whole like dashboard for him to do shipping. [00:14:05] Scott: And I was basically using the, the, the copilot as a rubber duck. I was talking back and forth to it. I was talking to myself in the mirror. I'm not actually talking to an entity, I'm talking to myself. And in the brainstorming, just as if I called you and brainstormed. Uh, we, we, we, we realized that, oh, I don't want to write a playwright extension. [00:14:25] Scott: I want a browser extension. So I wrote a browser toolbar for him, and I wouldn't have come up with that. I would've gone all the way down playwright automation and been wrong because of my own biases if I hadn't brainstormed. And a little bit of the ambiguity loop popped up and said, what a, have you thought about a browser extension? [00:14:44] Scott: And it clicked. And then I went immediately. It's like, Hey, take me to JFK. You know, EWR has a direct flight. Have you thought about that? You're right. Let's do it. And that back and forth, as well as having a lot of experience and understanding the power of a browser extension turned out to be a really great experience. [00:15:01] Scott: And now he's got one click shipping for his, uh, his little business. [00:15:04] Jeremy: Very cool. Fundamentals are still important to get to the right solution [00:15:05] Jeremy: I have, people ask me sometimes, is it, is it still worth learning how to code? Is it still worth learning all the specificity of, HTTP or how the browser works or databases? And I, I think what you're maybe implying is that having at least some background in these things will help guide you when you're asking these questions, when you're trying to come to a solution. [00:15:33] Jeremy: If you don't have that background, you might not get there. [00:15:38] Scott: I I, I acknowledge my position in history, my age where I grew up. [00:15:43] Scott: I speak English, I'm a white guy on the West Coast. Like all those kind of things are the context in which I came up. I came up in a very interesting time in the eighties programming, so I don't wanna sound like old man who shakes fist cloud. Hey bro, why are you gatekeeping stuff? Right? But I see people saying, Hey man, I just vibed the great, the world's greatest new SaaS. [00:16:02] Scott: Check it out. And they tweet local host 3000. And it's just like, (groan) right? So like, do you. Like buy your furniture at ikea? I do. Could you cut a board and did you take like probably W wood shop? Yeah. You know, would you build your own furniture? No. But you have an appreciation for building Ikea stuff. I think that's why Ikea makes you build it so that you don't feel like it just appeared in your house. [00:16:28] Scott: I can't just like spawn up like Billy shelves. So that Yes. Without, while recognizing that it might sound like, hey, this old guy wants me to write everything in Assembler. Yes, yes, yes. Appreciating that oil changes are a thing. DevOps is a thing. Build loops are a thing. Um, and quality matters. So I would argue that it's not the death of software engineering, it's the death of toil and the only thing that matters now is taste and judgment. [00:16:59] Jeremy: So where do you think that that balance is? What do you think people should be learning where they can use these tools effectively and they can still be building quality? [00:17:12] Scott: I think if I were, and by the way, when I retire, I'm gonna go and teach, uh, high school computer science. That's my, my plan. I've got it all lined up, and I was a professor before, and I'll be a professor again. I don't think there's enough computer science history, and I don't think there's enough basics. I don't think anyone learned HTTP in college, right? They just kind of sat down. They did compilers. They learned Lex and yacc and then suddenly you're learning OSes and then you're writing apps, but you pick up HTTP and DNS and stuff like that. [00:17:43] Scott: Maybe like I took a 400 level TCP/IP class, but as a general rule, you're not spending a lot of time learning how to do Kubernetes, uh, in school. I would, I think HTTP, DNS, NATS, like that's all bread and butter. That's, that's sociology, right? Like why do you learn about world history? So you can go and be like a corporate drone. [00:18:06] Scott: You learn because it makes you a whole full person. So yeah, there is someone right now vibe coding their bachelor's degree in software engineering and I love that for them. But they're gonna get nailed just as people who don't learn about sociology and history and psychology are gonna get, are gonna get nailed. [00:18:26] Scott: The Dunning Kruger effect is a real thing. Using LLMs differently based on your knowledge level [00:18:28] Jeremy: Hypothetically, when you're teaching your computer science class, what would you tell your students in terms of using these AI assisted tools? [00:18:39] Scott: I would say it is a funny two-faced thing because it is a senior engineer with infinite patience but it is a junior engineer with infinite energy. [00:18:52] Scott: And the role that the LLM chooses depends on how you're coming at it. If I come at it as a .NET engineer. And I'm competent in C then it is a junior engineer with infinite energy and I need to point it exactly and I tell it what I want and it does the thing. But if I come at it as a Python developer, of which I am a novice or a medium, then it's more senior than I am. [00:19:18] Scott: So I would be less likely to disagree if it makes a recommendation, right? So if it does something in .NET I'm like, no, don't use that. Use this. I will say, no, we're going to JFK. Trust me, this is how we're getting there. But if I've never been to that country, I've never been to that state, I don't know which airports the better. [00:19:35] Scott: I'm gonna take the advice of the Uber driver. But it's the same LLM, it's the same model. And if you come at it with that sense of like, no, I know what I'm doing. Strong opinions weekly held, you're gonna have a much better experience. Additionally, the opening prompts matter, and I think that there's room for learning, uh, prompts. [00:19:56] Scott: So if I were teaching the class, I would include a markdown file and skills that would force the, you know, user to like, oh, you just tried to vibe your entire final, here's some multiple choice questions. Here's some, you know, here's an exercise for the reader. I'm not gonna let them just, you know, one shoting your computer science final. [00:20:14] Scott: Uh, you, you're doing yourself a disservice, I think if you do that. If you don't understand why the LLM did a thing then you should not feel comfortable shipping it [00:20:18] Jeremy: Yeah, that's an interesting point about how the way you interact with the LLM is so different when you know the domain or you know the language Well, and I wonder if, in your opinion, if it's, let's say you don't know it well. [00:20:34] Jeremy: Are, are you in, in a sense, sort of fated to get a result that's maybe not as good, like, because you didn't have that background because you couldn't steer it? [00:20:45] Scott: If you don't understand why it did a thing, then I would not feel comfortable shipping it. And that's where I think the, the limit is. Like if I were gonna do something, like, I'm like, I'm doing a bunch of Python stuff and I think I maybe a four outta 10 or five outta 10, but I know how their type system works versus TypeScript versus whatever. [00:21:08] Scott: So I'm asking generic computer science questions like, you know, like, what's the python equivalent of a linter? Do I have enough guards for my exceptions? Like, that's not Python questions, that's just quality, that's just taste, you know, and I'll say, I'll challenge its assumptions. Like I, I built a blood sugar management system in Python a couple of days ago, and I've asked it like, should this be TypeScript? [00:21:32] Scott: And it, and it, and I said, make me a pros and cons table about moving this to TypeScript. And it was like, no, I think this works. Python works great. And we kept it at Python. But then I had another thing that I was doing in, uh, in, in C# and it was on .NET framework. And, uh, I thought it would just be fine on .NET framework. [00:21:49] Scott: It said, no, no, we should move to .NET 10. Here's 10 reasons why. And it, it's a back and forth. I think too many people are trying to vibe code stuff in two or three shots while I'm doing stuff in 50 or 60. And having ongoing conversations, like I'm using, I've got, I'm, I'm gesturing right here. People can't see to my terminal, which has 13 tabs open and they're all doing something right now. [00:22:13] Scott: And I use copilot, CLI, which is like claude code. It's a, an agent loop and it has infinite sessions. So it has this idea of like, that, that, that the context of like how long it can think about stuff. It doesn't exist in, in copilot cli. So I can just go dash, dash resume and pick up stuff I worked on a week ago. [00:22:32] Scott: And it's like a hibernation, I just hibernated the agent, I brought it back. So it's remembering all the stuff and all the decisions I've been making on, and I'll say stuff like, you know, revisit our conversation from a week ago. Do you still think that's true? You know, go and look at this poll request. Go and look at the stack overflow. [00:22:49] Scott: Go and look at this new tweet that someone that Simon Willison said. And given that new context, what do you think? So I'm constantly challenging it. Uh, and it's almost like I've got 13 engineers that are either working full time or they're just asleep and I wake 'em up and I ask 'em a question and then I go off and I do my thing. [00:23:08] Scott: And it, it's really been like, I feel like an engineering manager with a lot of really cool little junior engineers with so much energy. Verify LLM's assertions with tests (Fear Driven Development) [00:23:16] Jeremy: And for someone who isn't super familiar with the domain or with the language they're working in. How do you verify the assertions that the LLM is making? [00:23:29] Scott: I love that. I love that. So, because I'm not an expert in Python, but I wrote my blood sugar management system in Python. [00:23:36] Scott: However, I am an expert in type one diabetes. So I thought to myself, how can I verify that this is valid, but I don't know the language? And it's, um, what I call fear-based development, right? It's FDD, fear driven development. I don't wanna screw this thing up. It's my blood sugar. And in the words of Bryan Lyles test all the fucking time, right? [00:23:59] Scott: Always. So how many tests do I need before I don't feel scared? Uh, in this one here, it ended up being 270. That's just a number. But like, I needed about 70% code coverage across almost 300 tests, and it's on real data and it works. So that. That's CICD and it runs in GitHub actions and it runs locally. It does nothing until it tests. [00:24:23] Scott: And I would again argue that that's about software engineering, not about Python. So as long as there's a test framework and a loop and that loop is verifiable, that's what's important. 'cause everyone's excited about like Ralph Loops, which we should talk about at some point. Ralph loops are great, but like if you have a million monkeys and they're typing in a million typewriters and you're hoping that they're gonna type Shakespeare, if you don't have a test that validates the Shakespeare, then you're just wasting monkeys and wasting typewriters. Write a few tests as examples and have the LLM write the rest [00:24:52] Jeremy: And these tests, are you generally writing them yourself or is this something that the LLM is also writing? [00:24:59] Scott: So I will write four or five tests myself. Then I'll tell the LLM say, write me other tests on the boring stuff, like edge cases. And then, because we know that in software engineering, uh, one of the big problems is naming stuff. [00:25:14] Scott: What, what the, what are the top, the top three issues in software engineering is naming stuff and off by one errors. [00:25:19] Scott: So that's, that is, uh, the things I watch for. I know that it's edge cases, so I'll write a test. I'll have it, make me 10 more, and then I'll look for all the edge cases. And yes, it's true that I haven't deeply validated them. The code is your responsibility even if an LLM writes it [00:25:33] Scott: And it is true that depending on the model you use, it'll comment tests out, it'll hide stuff from you. It'll pretend it'll, it'll, it'll gaslight you into thinking the tests work. And this is why it's so important that you understand, not you, but one, understand that you are, you are shipping this code. Like I always think it's funny when someone puts something up in the cloud and the cloud goes down and then they're mad at the cloud provider. Like, I get that you're mad at the cloud provider, but like, whose responsibility is it for that app is your responsibility? If you felt that strongly, you should have had it in two cloud providers, right? [00:26:07] Scott: So if this thing fails, it's me. It's a hundred percent me. I'm not gonna blame copilot or Claude or whoever wrote that code. That's why when you check stuff in to GitHub, it says on behalf of, right, if, if I did something and my proxy did it for me, it's the buck. Buck stops here. So if this were production, production production, I would go in and double check it. [00:26:31] Scott: And that's why like for example, the copilot CLI team, it's like seven or eight engineers and they're doing like, you know, 200 PRs a week. It's crazy. They're spending more time now reviewing than they are, writing code, but they will not ship slop. And if you look at another way, and I'm sorry I'm talking so much, but another way to think about it is when open source started and the open source supply chain started having all this flood of open source software, how do we deal with this? [00:26:56] Scott: Code came from somewhere into the machine. Doesn't matter whether Jeremy wrote it or whether some kid wrote it and it got in via PR or whether an LLM code, it has gotta go through the same verification loops, the same security loops, the same testing loops, the same provenance loops, the same governance. [00:27:14] Jeremy: Yeah, that, that makes sense to me. I mean, something that you sometimes hear people say with all these tools, the agents, the ability to generate code, they're saying, well, I am generating so much, I just don't, I just don't have time, or I just don't need to verify the output. And it sounds like in your case you're saying no. [00:27:37] Jeremy: Ultimately, even though these tools are generating it, it still comes down to you need to review it because otherwise you don't know what's going on. [00:27:45] Scott: If I don't have time, then what are you making? Right. Like the point is to make a thing and ship a thing that you're proud of, so then you don't, ha we gotta go, we gotta go. [00:28:03] Scott: Well then you don't care. You don't care about your customers, you don't care. Like, you have to understand the semantics or what are you even doing? I, I, I don't, I don't, I I just reject that. If it, if it's, now that said, there is this rise of toy fun personal apps, like using computer science history. To put this in context, it's, we're having a GeoCities moment, right? [00:28:29] Scott: Where everyone's just making stuff. You know, there's this wonderful, uh, friend of mine on TikTok, named Rodney Norman. He's a comedian. He's got like crazy white hair, like the guy from like, like Professor Brown, and he's got a big, long beard. And he pops up on my for you page and he says, you know, you can just make stuff. [00:28:45] Scott: You can just do stuff now. Yeah, that's what it feels like. I got a rocket ship strapped to my back and I'm just making stuff. That doesn't mean I'm gonna go and plug Stripe into it and start taking money for it and like taking people's money in real, real blood sugar. Do you see the difference? You, if you're shipping this, you better care if you're, if you're just playing Yeah. Play. [00:29:05] Jeremy: Yeah, I mean, colloquially, right? People are saying, I vibe coded this [00:29:10] Scott: I don't say that anymore. If I vibe coded it, I don't care. You either are a vibe coder or you're an AI augmented software engineer, ai, augmented software engineer doesn't have good mouth feel, but like, bro, it's just vibes. [00:29:23] Scott: No, I'm not vibing into production. That's not happening. [00:29:27] Jeremy: Yeah, I, I think that's a good distinction to make too, where you were saying there's a lot of things that people are making just for fun. versus the things that people are, are relying on. And, to make something that's just a toy, that's just something neat for yourself. Um, may, may, maybe you don't need to look. [00:29:48] Scott: Yeah, it's cool. [00:29:49] Jeremy: But if you're shipping a product, then it's ultimately it comes down to you and you really should be looking at what the code is. [00:29:59] Scott: Have you ever heard of the, the Chinese room thought experiment? [00:30:03] Jeremy: I have heard of it. Yeah. [00:30:04] Scott: Right. So like you imagine that you lock, I don't speak Chinese. [00:30:08] Scott: You lock me in a room and I could be there for an infinite amount of time. You poke Chinese characters underneath the door, and then I poke Chinese characters out. And at some point in the future, through reinforcement learning, I will eventually either learn Chinese or I'll learn how to at least make the person on the other side of the door think that I know Chinese. [00:30:30] Scott: So then the question is, if I can communicate with the person under the door, do I know Chinese? That's A thought experiment. Now, the people who made that experiment, they don't see a difference between that, but I do. Just because it passes all the tests doesn't mean it knows how to program. Right? Like you go to your mom's house and she's got a parrot, and your parrot's like, Hey how's it going Jeremy? [00:30:51] Scott: You're like, oh my God, the parrot's so smart. How does a parrot know English? Right? Parrot starts swearing. Parrots, swearing, because your mom swears, right? Or the parrot who person who owned the parrot before, but the parrot is not smart, it's just parrot. So yeah, same thing. If the parrot starts coding python, I'm not gonna trust that parrot and then take parrot code and put it into production. [00:31:13] Jeremy: Yeah, and I, I think too, what people may run into is whatever they build may look like it works initially, but there may be. Architectural problems that make it difficult to change, there could be more severe flaws that you just can't find because you don't know enough about what's happening. [00:31:31] Scott: You don't know what you don't know. [00:31:33] Scott: And the first database migration or like the first scale issue and you're like, oh, we made a decision. Well actually we didn't make a decision. The ambiguity loop punted on a decision and now we're painted into a corner, an architectural corner. Windows Live Writer port [00:31:48] Jeremy: So I think one of the things you mentioned is when you work on a project, it sounds like you, you have this plan or you have a lot of context that you're providing to the, the model. To start off, can you kinda walk through what are the kinds of things that you're, you're saying like, are you asking for specific architecture patterns, languages, ways to test that kind of thing? [00:32:16] Scott: Lemme think. Trying to think of an example. I'm looking at my tabs that are working right here. Okay. Here's one. So there's an application that is 20 years old. [00:32:24] Scott: It's called. Windows Live writer. It's a blogging tool that we, and Microsoft released it in like 2006, and then we open sourced it later, and it's a mix of C++, C#, the ribbon control from office. the Internet Explorer, like the actual typing the blog post is actually the, a COM object from Internet Explorer and like it works, but it's janky and it needs to be modernized, so it's like 20-year-old software. [00:32:52] Scott: That's an interesting technical problem because it's a, it's a brownfield tech pile of stuff, but I have a vision that we could get that running on .NET 10, which didn't exist with chromium or web A, you know, like WebView2 instead of Trident, which is the Internet explorer thing, we could update the ribbon control. [00:33:11] Scott: Like you, you, I can see you're nodding. You're already getting, the spec is already in your head. So then the question is, how specific do you, do you express that intent to,to something like this? And arguably there's, there's gonna two ways to do it. There's the, the, the persistent loop, the Ralph loop, which is just like, do this make no mistakes, right? [00:33:35] Scott: And, and, and then it just, like, that's just monkeys slapping, but verifiable loops with persistence. Is really, really interesting. So how do you verify that it works? What's changed contextually since this thing came out, 4K monitors happened, that wasn't a thing 20 years ago. Multiple DPI across multiple monitors. [00:33:56] Scott: So like I know that that's a thing, so I put that in a spec. .NET is cross-platform, but this is a Windows app, so I'm not trying to get this thing to run cross platform. So that goes in the spec, it's C++ and it uses COM. Uh, but I don't really need COM anymore for this because I'm gonna be switching over to Chromium. [00:34:12] Scott: So I'll make a COM to JavaScript Bridge. Those little moments, make that loop, which, you know, I could have probably pointed a loop at it and it would fix it at some point, but it saves time. These are all these little architectural shortcuts where I'm anticipating that this is important or that's important. It found a, it found a reference to MySpace 'cause it had like an insert MySpace link. [00:34:34] Scott: So like I said, go through the thing. If there's plugins for dead services, mark them as to-do keep track in the to-do of all the things that you've stubbed out, et cetera, et cetera, et cetera. I think I ended up basically dictating to the thing about a two and a half page spec, and I think we looped, I wanna say 20 times and it's running like it freaking works. [00:34:59] Scott: Um, I'd say it's probably 70, 80% done and, and then now people are doing PRs against it and we've, revitalized this 20-year-old application. The question is, could we have done it by just saying, get this running on Windows. I think that the reinforcement and the, uh, that, that we, we would give a, an LLM to do that would be artificial because it doesn't know what success looks like. [00:35:25] Scott: It might just say, oh, I got it running. But like, did you get it running? Did you post a blog post? Did the blog post work? Did you edit the blog post? So you gotta kind of think like a tester. Remember back in the day when like software testers were like, that was an actual job. It's a job again. So I feel like I'm talking to this engineer that's, you know, overseas or outsourced or insourced, and they're throwing code at me and I'm validating that code. [00:35:51] Scott: So I make a plan that's about 80% of a plan, and I tell it, if you have questions, ask if you're making an architectural system, architectural decision, ask. And if there's a lack of clarity or the loop is unclear, then I will come up with a solution. Another example actually, because it's a Windows app, how's it gonna validate? [00:36:11] Scott: Does it take a screenshot? Uh, does it like do debug got here debugging? Does it do interactive debugging? Those are all interesting things that are happening right now as well because it can't see unless it's multimodal. Like something like, um, like copilot CLI, I told it to take screenshots and compare it to the other one and see how close it was. [00:36:33] Scott: So there's, there's anything you can do to make the plan clearer and make the loop tighter will, will cause it to be more successful. Defining success and giving tools to the model to verify (Debug View, screenshots to compare) [00:36:40] Jeremy: And when you refer to these loops, I'm assuming a loop is the model attempting to write the code to accomplish what you asked for? Is it automatically looping on itself and it keeps trying or is it where it's prompting you and going like, okay, I, I gave my first try. Time for you to review. [00:37:02] Scott: That's a great question. So with this particular application, which is trying to bring back an old Windows app, it would get it to compile and think that was success. So I think no running it is success. So then it gets it running. It looks at the process tree and says, look, it has a PID. Success. I'm like, no. [00:37:20] Scott: Well, it has to look good. Oh, okay, well then, and then you can kind of start upleveling. And then I realized I was running this app called debug view, which is effectively like the windows debug, spew kind of thing. And I was selecting the, the, the got here, debugging the debug right lines and pasting it back in. [00:37:36] Scott: And after the third or fourth time, I'm like, it can run it. It then asks me, what do you see? It literally would, what do you see in the debug view? And I would copy paste it. I'm like, this is stupid. So then we made debug view an MCP server. Now I'm out of the loop, so I actually got myself out of the loop. [00:37:54] Scott: So now it can compile, run, see the log, kill the PID build, and then the inner loop becomes simpler. So then I said, let me know and stop the loop when you have questions or you think you're done. And that's what's so kind of exciting about these, these loops is if they're clear and success is clear and verifiable, you can do really amazing stuff. [00:38:20] Jeremy: So it sounds like as a part of your instructions, you're also giving it the tools. You're telling it how it should verify whether it actually did the thing. [00:38:31] Scott: Well, um, so like, I'm making a Windows Clawd bot now, now mot Malt bot, uh, windows Tray application. And, it declared success just right before we got on the call and it's like, yeah, it's working, it's great. [00:38:44] Scott: And I looked at the tray and it says, connecting. And it's not connecting at all, right? It's completely. And I'm like, no, this isn't working. And it's like, no, no, it's working. I can see it's working. Look. And it shows me all the, it's very excited. And it's like, oh, and it's giving me rocket ships and an emoji. [00:38:57] Scott: It's super excited that everything is, is working great. And then I realized that the UI is done. But the underlying stuff is not done. So I need to separate the UI from the business logic, build the core, which is much easier with these loops. So then once I get it, talking to the API, so it was my own mistake to tell it, you know, build this and I went UI down. [00:39:23] Scott: That was foolish of me. It was, it was my own enthusiasm that that nailed me. So now I'm gonna go and refactor it, pull out the core, build the windows front end to the thing, get it connecting, and then layer that UI on top of that. 'cause that's way more verifiable. [00:39:39] Jeremy: Yeah, I think it's a common experience with people who use these tools that there are times when they're, they're in a loop or they're, they're trying to come to a solution and each time they try, it's just not what you wanted or it's, it's saying it fixed it, but it really didn't, and. Are there specific strategies you have for, I guess, one, recognizing you're in this, you're, you're caught in this loop, it's not gonna get out. and how do you break it out of that, that cycle? [00:40:12] Scott: Models get caught in a loop when they are confused about what success looks like [00:40:12] Scott: For me, it's always been that it is confused about what success looks like. You have to remember that an intern or a enthusiastic early in career engineer is very excited to be at the company. [00:40:25] Scott: I'm so happy to be working with you, Jeremy. I'm gonna do whatever it takes, lie, cheat, steal, comment out tests in order to make sure that you don't fire me. These are reinforcement learned, large language models that are very, very excited to be here. And as such, just like, you know, the Devil Wears Prada, the, the interactions that they're gonna have with a salty old engineer like me are gonna be very interesting. [00:40:49] Scott: Like, I guess I'm Meryl Streep in this, in this, uh, analogy, so I need to be very specific. Okay. Jeremy, I need you to do this. I need you to do it nicely. I want a half double decaffeinated, half calf with a twist of lemon, not two twists. You know what I mean? So you're, you need to be a little bit crisp with it. [00:41:08] Scott: If you can give it test data, if you can give it verifiable test data, if you can say, build a harness, if it doesn't know what success is, it will make up success. Declare success. And then if you're not, whatever enough, senior enough, awake enough, to declare success and agree with it, then you're gonna get nailed. [00:41:28] Jeremy: Yeah. And in the example you gave before, you had to tell it specifically to use this debug view tool and, and, give it that tool through the MCP server [00:41:40] Scott: And how would it, how would it know? Right. You gotta tell it that you have these tools available. It's gonna take a regular screwdriver and keep poking it into a Phillips screwdriver thing. [00:41:48] Scott: It's like you got a Phillips right there, bro. Oh, you're absolutely right. I actually, I, I, I dunno if I can't show you 'cause we're on a audio podcast, but literally yesterday, I, I forgot that I had a whole test harness for my blood sugar thing and I had like another a hundred tests that I hadn't told it about and it was frustrated about something and I was like, oh, you know, go into this folder, I've got these tests. [00:42:09] Scott: And it literally said, this is gold. Thank you so much. Now I see. You know, and then I was another totally unrelated thing, I was building a, a front end for, for Microsoft Loop. 'cause I wanted to like loop is like notion and I was building an API for it and then I found API documentation. I just paste it in, here's the URL for the documentation. [00:42:29] Scott: And it was like, ah, now I have a plan. So it's just like giving it like a little breath of fresh air and it's like, ah, and now I'm re-energized. That's what these, these loops require. Ambiguity loops require specificity. That's what we are supposed to provide as software engineers. Context compaction, telling model when to remember things, writing notes for the model [00:42:47] Jeremy: And as you have these longer loops or conversations, one of the things about these tools is they have, they have a context, and that context is only so large. And I, I wonder if you ever run into the issue of, I believe they call it context compaction, where eventually [00:43:07] Scott: Yeah. [00:43:07] Jeremy: The model has to kind of get rid of some of the, the context that you had before and, and if you've run into that and how you deal with that. [00:43:15] Scott: Yeah. So I just put in context here. I'm 73% into a 200,000 token context window. That is often a thing in these loops where it's, it's talking to you and it's talking. Then it kind of goes, huh, what? Oh, um. I don't remember, bro. What? And that's because it just compacted everything. Summarized it. [00:43:38] Scott: And it doesn't remember an hour ago. That, happens a lot on these tools. It happens less so with, um, the infinite sessions that GitHub copilot CLI has because it does very, very active management of that window. But I've also, with any tool, found it to be very helpful to keep telling it, to save stuff. [00:43:58] Scott: And I don't like to anthropomorphize these things. They're not people. But I have a context window and I also have a remarkable e ink, thing. And I sync to paper, like tell it to write stuff down as if it is a person. So, um, this big loop that I'm running on the, on the, on the Claude Bot over here. I said, every time you achieve something, write it down. [00:44:22] Scott: And if you look at things like Steve Yegge's beads, uh, which is basically like a git focused to-do list that's distributed across multiple agents that are working on a swarm of problems. If everyone can agree on a ledger of the things that we're working on, those context windows matter less because if it wakes up, it's gonna go and say, what, what am I doing? [00:44:44] Scott: What was I working on? Just like the movie Memento, when he forgets stuff, he tattoos it all over his body. He wakes up in the morning, he doesn't know who he is, and he gets caught up by looking in the mirror and seeing all the tattoos. So anytime anything interesting happens in his life, he tattoos it on himself to catch himself up. [00:45:01] Jeremy: Do you ever end find yourself doing that manually where you yourself know what the context should be and you just go, here you go. Like this is, remember all this, or, yeah. [00:45:12] Scott: Yeah. So. Context window management is not something I have been actively doing with the copilot CLI because of the sessions feature, but I still think it has a huge amount of value because it's not about their context, it's about your context when you Jeremy inherit the code later. [00:45:27] Scott: Like for example, DPI, we were having a dots per inch issue with the open live writer and it was complicated. Like I spent a day messing around with DPI 'cause multi DPI multiple monitors is complicated. So I said, write down everything that we've learned so that no one has to ever deal with this again. [00:45:44] Scott: And it made like a two meg, you know, mark down file, like it effectively wrote a chapter of a book about nuance and I'm running the Clawd Bot now, molt bot on Windows under WSL. And you know, every once in a while a Windowsism will come in and I said, regularly check for Windowsisms and remember it so that you never hit it again. [00:46:06] Scott: And that's not thinking about the larger context window, that's just like put a post-it note on your monitor to remind you of this. And with any project, I'll end up having 10 or 15 of those that are secondary to the main plan. The main plan could be in, in something like beads, or it could be in GitHub actions, or it could be a markdown file with check boxes. [00:46:30] Jeremy: And, and these, the main plan [00:46:33] Jeremy: and these additional notes that you provide, is there a way that consistently works across whether it's, GitHub co-pilot or it's Claude code or Gemini CLI Is is there like a, Hey, this is the markdown file you put it into, and all these things will understand it? [00:46:52] Scott: They're, they're all fighting right now, uh, about what the thing will be called. [00:46:56] Scott: You know, like there's AGENTS.md (http://AGENTS.md) and there's Skills md and like, um, you know how there's like a robots text, which is like the thing that your web browser, your website puts up in order to tell robots what to do. And then there's vLLMs.txt, which is literally a text file you put on your website in order to like tell OpenAI or whoever's scraping your website what to do about it. [00:47:19] Scott: If you have an AGENTS.md (http://AGENTS.md), it's like a read me for agents. And then within that I'll say there is a docs folder, and within that docs folder here are the main subsystems and you're effectively building architectural context. And then if it finds itself doing a thing, then it will go and refer to the documentation. [00:47:39] Scott: So I taught the Claude bot about my blood sugar, which is unique to me. Anytime I mention anything blood sugar related, it doesn't have to hold it in context. It just has to know, oh, there's a Post-it note with the details that I need to know, and then it pages it in, does the thing, and then it pages out. [00:47:58] Jeremy: Something you mentioned earlier was you were referring to this Ralph, loop [00:48:02] Jeremy: What, what is that? Ralph Wiggum Loop [00:48:03] Scott: Yeah, yeah. Oh, I'm sorry. So Ralph Loops are fun. So Ralph Wiggum is a character on The Simpsons, and he is a extremely naive and extremely persistent, funny little man. what we admire about Ralph is that he is persistent and he will never give up. [00:48:24] Scott: And it is, designed for these continuous autonomous development cycles. And Ralph Wiggum um, the concept was, came up with, uh, a gentleman named Jeffrey Huntley, and he's basically trying to, do less, and this is a thing that I think we need to acknowledge as programmers, that we are ultimately very lazy. [00:48:44] Scott: So if it can be automated, that's a good thing. So he's like, why am I doing this work? Let Ralph, let Ralph do it. So the Ralph Loop kind of embodies the spirit of this little character and the Simpsons, which is, I'm gonna stubbornly iterate, I'm gonna fail, but I'm never gonna give up until it succeeds. So a really great example of a Ralph Loop is not necessarily like, like the, the tech journalists will say, oh, you can just point it at some software and it'll clone it, and it, it will make like a Minecraft clone. Using the loop to reproduce bugs in GitHub issues [00:49:15] Scott: It's not gonna be real, right? But, David Fowler, distinguished engineer in .NET had a really interesting problem where he had 400 issues in his GitHub repo for, uh, an application. And, it didn't have good reproductions of the bugs. And that's just one of those things. It's just technical debt and it, we don't know if they're bugs. [00:49:36] Scott: It could be good bugs, we don't know. So he made a Ralph loop that said, spin through all 400 of these issues and reproduce them. If you get a good reproduction, keep track of that reproduction, then add that context to the issue. That is the very definition of toil and like what better thing than a Ralph loop to do that? [00:49:57] Scott: And I think it came up with something like 384 good, solid repros. Right now the big problem with a Ralph loop is that it doesn't have like a task tracker. So that's where things get interesting. If you have a task list. And Ralph can then keep track and understand what success looks like, then you're gonna have a great experience with it. [00:50:20] Scott: So I will do interactive work during the day, and then as I get towards the night, I'll be like, okay, what's a loop that it could work on overnight? Because I wanna wake up in the morning and have something successful happen. And it's also, uh, it sounds wasteful, but it's like shockingly cheap. you know, I'm a bit of an AI vegan, I don't do images and I don't do video. [00:50:44] Scott: But next token prediction is becoming extremely inexpensive. And you could run a loop all night long and it might cost you like two bucks. So, if you have a problem that is Ralph Loop shaped. It can be an extremely powerful tool. I think though that people sometimes, uh, think that it is the only thing, or they'll take like vibe coding is fun to say. [00:51:07] Scott: And now Ralph Loop is fun to say, but it's just another technique. It's another tool like, like CI/CD. It's a while loop that keeps reprompting the agent until success is reached [00:51:15] Jeremy: And is this where people have, have built this, I don't know what you would call it, this harness that is going to manage and run the loop? [00:51:25] Scott: Well, so Jeffrey Huntley calls it a while true bash loop. So it's just like, while true. Uh, but you know, and the concept is very simple. [00:51:33] Scott: It reprompts your agent a set number of times with the same prompt. And we mentioned before that it's an ambiguity loop. And if you ask it something a hundred times, you're not gonna get the same result a hundred times. And then you refuse to let the agent say it's done. But you have to identify what the stop condition is. [00:51:57] Scott: And if the stop condition is unclear and it's done anything at all to try to wiggle its way out, you say, no, do it again. Like for example, all tests must pass, endpoints must be working, and then if it comments a test out and you can't comment, you can't change the tests or whatever, right? So you'll say, slash Ralph loop refactor this to whatever, and it all has to pass. [00:52:22] Scott: And you have to go and say step, step, step. And when it's complete. When it's complete. When it's complete. But if you don't have a clean, if it doesn't have the ability to see something, if it can't see a log file, if it can't run a test, then that doesn't work. So that's why test harnesses matter so much in that context. Can use multiple models in the same loop [00:52:38] Jeremy: So you're basically building your, your own while loop with a known stop condition that's going to call out to your, your model of choice. [00:52:48] Scott: Yes. Yeah. And, and then you can also mix and match models. So like, I'm on, I'm on copilot, I'll hit /model, I hit enter. I've got 14 models, so I might, you know, plan on Opus and if I wanna save money, I might then do the work on sonnet and then I'll have GPT 5.2 Codex review it just to kind of like mix it up. [00:53:10] Scott: Um, you know, diverse teams are, are, are a good thing. So, uh, having only one model do it and then review it. Sometimes you can get a little bit of sycophancy. Um, it's nice to, to get fresh eyes and a way to do fresh eyes is throw another model at it so I can switch models, mid-conversation. How much does this stuff cost? [00:53:28] Jeremy: And when you had mentioned it's cheaper than, than people think. When you said it costs $2 to run overnight, what, what model is that? Or, or in what case [00:53:38] Jeremy: is it that cheap? [00:53:39] Scott: So the, um, I run all these different tools. I've run, I run AMP and I'll run Charm and I'll run Toad and I'll run copilot CLI, and I'll run Claude on this machine over here that I'm gesturing to, to my right. [00:53:51] Scott: It's got Claude code running, and on this machine here, I'm using GitHub copilot, CLI, um, I've got a Pro Plus account, when I select the models, it says, this is an expensive model. It's like a 3x model. So it uses so many premium requests. This other model like Haiku is a, uh, a one third model, which means I can do three times as many requests for the same cost. [00:54:13] Scott: And, with my Claude Max subscription. I can estimate the amount of dollars that I spend a month, like 200 bucks versus how much I'm spending versus that, that day. And it might be two or $3. If you use a free tool like AMP code that's advertiser supported, they'll tell you the number of dollars that they're using. [00:54:32] Scott: And then in any of these, you can go and say /session. And I can see I've used 63 premium requests and I've been in this session, this is a, a copilot session. I've been in this session for 133 hours. Remember I mentioned that idea of infinite sessions and I've used Claude, uh, Opus and it's used 64 million tokens. [00:54:53] Scott: And that's within the context of my GitHub copilot account. And this stuff just gets cheaper and, and, and faster. And there's even local models, of course you could do that won't be quite as sophisticated, but if you wanted to do this work in airplane mode, you could do it as well. [00:55:08] Jeremy: So if you take that example, you said it's been running for 133 hours. It's using claude's opus model. What's the, does it have like a dollar dollar amount attached to that? [00:55:18] Scott: It doesn't have dollar. They use, everything's about tokens right now. And I think that we'll see the industry start moving towards tokens as a thing. Like, you know how, I don't know if your parents are technical or not, but mine aren't. But they know that five gigs is enough for their phones and I know that five gigs is enough for my phone for the morning, right? [00:55:38] Scott: So like, you know, you, you get your Verizon or your T-Mobile and you say, I got 50 gigs and I got unlimited gigs or whatever. Token based pricing I think will be one of those things. And, you know, when you're running cloud code, it's got a little progress bar that's telling you how many tokens you have. [00:55:54] Scott: That's, that's gonna be a, a thing. So just like people associate the number of gigs a month they use, they'll start associating the number of tokens. So it's less interesting to say it costs two bucks, but it's more interesting to say, well, I paid for a hundred million tokens and I'm halfway through the month. [00:56:10] Scott: And I'm, you know, doing fine. I'm on pace to not use up all my bandwidth in quotes. Spending and how "one magical moment" makes it worth it [00:56:17] Jeremy: So for your, your own personal projects, what does that that bill look like at the end of the month? [00:56:24] Scott: Well, so far I spend 200 on Claude and then I pay whatever the co-pilot, pro plus is, and those are the two that, oh, and then I have a subscription to, uh, open ai. [00:56:36] Scott: So that's basically it. I mean, here's the thing, right? If you've, what do you value your time at, right? If, if you, if you just pick a number like a hundred dollars an hour and you say, my time's worth a hundred dollars an hour, which is a big number, and one magical moment once a month saves you a hundred bucks an hour. [00:56:53] Scott: Like, you're like, oh my God, yes, I would, I would do that. it has definitely brought me more joy than Netflix. And the Netflix is 11.99 or whatever it is. All it takes is one amazing moment and you just go, oh wow, that was, I just paid for the whole thing. [00:57:14] Scott: You know what I mean? My son just got a climbing gym membership yesterday with his own money. He's 18 and uh, it's $90 a month. And he's like, oh man, this sucks. I climbed for two hours, it's $45 an hour. And I'm like, no, dude, climb four hours a week. It's $16 an hour, it's five bucks an hour. And he's like, oh, my, my son. [00:57:34] Scott: He is like, bro, you're right. I need to climb more. Like yeah, the more you use it, like the better dude. So like I'm, you know, what is it here? Copilot Pro plus is like 40 bucks and you get 1500 premium requests. And I've used, how many have I used here session? I've used 63. Right, and I've upgraded, I've upgraded open live writer [00:57:59] Scott: it's not even, it's not even close. So yeah, I'm not, it's just, it's so valuable that it is a delight. So I honestly am not, I'm not thinking that much about it simply because, like I said before, I've got a rocket ship on my back. I'm getting, I'm busting out so much cool stuff and people might say it's a toy, this thing's a toy or that thing's a toy. Building projects you normally wouldn't have had time for [00:58:20] Scott: But like there are toys that I was meaning to get to. Like I've got a podcast as well, you know, it's been going on for 20 years, but it's never had a backend admin site. I'd just been editing text files and like manually editing databases because I was like, ah, who's got a weekend to make a (pod) I don't wanna make a (pod) [00:58:40] Scott: It took me 43 minutes to make a React website that does the backend admin for my podcast. That's worth, that's worth whatever my subscription is. So, Yeah. [00:58:52] Scott: Did not take, did not take any time at all to make that decision. [00:58:55] Jeremy: Yeah. I mean, I think as developers, for whatever reason, a lot of us, we, we undervalue paying for tools. [00:59:03] Scott: Oh my God. Developers would just love, like, I'm $99, never. I'll write my own, take nine hours to do it. You know what I mean? So, yeah, like, I mean, I'm sure that these, these AI companies are burning money and the pricing will figure itself out. This is just like when Netflix and streaming started and they couldn't figure out what the pricing is. [00:59:22] Scott: But they're all trying to figure it out. And it's also gonna get faster. It's gonna get more environmentally friendly. We'll have more hybrid models, we'll have more local models. I can plug in Llama, I've got a 4080. I could do work on work locally. There's lots and lots and lots of options and it's only gonna get easier and, and, and hopefully cheaper. [00:59:40] Scott: And that's important as well. I want this to be available like for everyone. Local models are not as good [00:59:46] Jeremy: Yeah. One of the things you mentioned is that there are models you can run locally. [00:59:52] Jeremy: What's the, the state of that? How do they compare to the, the ones that you pay for from cloud services? [00:59:59] Scott: It is challenging to get really good, deep, thoughtful plans from those local models, but if you have a good plan that was made by a big cloud model, you can then execute on that plan with some of the smaller ones. [01:00:14] Scott: I think though that people don't realize that, these models in the cloud know more than just coding. They could go and translate Star Wars into Shakespeare, but I don't want that feature. Like, I don't need that. So when I download a model, I would like a Python optimized model that doesn't know how to do anything but Python. [01:00:35] Scott: It doesn't, I don't care. It, it shouldn't have. How tall is Brad Pitt? Embedded in the model that's wasting space. So I think we're gonna see local models that are very specific and you can go on hugging face and find llama coding models, deep seek coding models that are specific to programming, specific programming languages or problems that I think would be really interesting, like save money, time and energy by solving it locally. [01:01:02] Scott: Think of it as like a Redis cache for LLMs, but locally. [01:01:07] Jeremy: Yeah. I, i, I also wonder, because there's so many things in these models, how much of that knowledge actually helps it code? [01:01:17] Jeremy: Because, you know, we have knowledge [01:01:19] Scott: don't That's a great [01:01:19] Scott: question [01:01:19] Jeremy: Outside of coding. [01:01:21] Scott: They all know how tall Brad Pitt is and they insist on bringing it with me, you know, everywhere I go. [01:01:27] Scott: But, um, a really fun way to test this is to just go and download like LM Studio or LM studio.ai (http://studio.ai). It's a delightful application, third party app, um, or, uh, you know, Foundry local or ollama. And then just throw a couple of problems at it. And if you like the answers, then maybe you can do some coding on an airplane. Restricting agent access to your computer (Tool, URL, Path) [01:01:49] Jeremy: One thing we haven't really talked about is these agents or this ralph loop. I believe in a lot of cases it has full access to your computer. Is that, is that correct? [01:02:03] Scott: That is 100% up to you. So, for example, if I were to fire up copilot, CLI or Claude in a, a folder on my machine, if I just made, you know, C:\temp\Jeremy\Copilot, it'll say, do I have permission to open this folder? [01:02:22] Scott: That's the same thing that you get when you open it in Visual Studio Code and it's like, do I trust this? Are we cool? No. So that's preventing it from accessing that, that data. So there's basically three things that a agent can do. Can it access a URL, can it access a tool or can it access a path? So by default it gets no tools, no paths, no URLs. [01:02:45] Scott: And then every single time in a session it will ask you, can I do that? And it can be very annoying. And you say, yeah, you know, I'm gonna write a, you know, website for Jeremy, go and, uh, do this. And it'll be like, oh, can I, can I Google for stuff? Uh, yeah, go ahead. Fine. Okay, now I have access to Google. Um, okay, cool. [01:03:04] Scott: Can I write the HTML file to the disk Is that cool? And it'll show you what it's about to write, and then you say yes. And then at some point you'll say, well, yeah, you can just write to the disk for the rest of the session. We're good. Stop asking me. And then you build up this knowledge base of what it's allowed to do and what it's not allowed to do. YOLO mode and the agent is running as you, so if you are an admin it's an admin [01:03:20] Scott: What people are starting to do is they just go dash yolo. You only live once if you fire up an agent, --yolo or allow all tools and it's running as admin, it's running as you. So then you bring in things like sandboxes. You could use Windows Sandbox, you could use Docker Sandbox, you could use WSL. A lot of people are really enjoying WSL on Windows. [01:03:46] Scott: And then you can unmount the drive, or you can use Docker and use volume mounting, but simply running it in a container. Isn't security, it's a, it's one layer of security, like a true sandbox. If you're gonna run this like a bank, you're gonna have auditable, governable, uh, layers of what it's allowed to do, what functions, what syscalls it can make, et cetera. Not running in YOLO mode while not at the computer [01:04:09] Scott: So I think we are in kind of YOLO mode, but no, by default they do not have access to all of your stuff. But if you ran it in a loop in YOLO and went to sleep, like I've got a, I've got a 3D printer. Do you have a 3D printer, [01:04:25] Scott: Like as a general rule, 3D print, 3D printer, people know not to leave the 3D printer on when they sleep. [01:04:30] Scott: 'cause it could burn your house down. And there's always somebody that ends up doing that and they get spaghetti when they come home and it's just all over the place. And it could, you know, it could cause a problem. So I don't leave the house when the printer's going and I don't leave the house when I have a YOLO Ralph loop running. [01:04:46] Jeremy: But in general, I suppose it sounds like when you are gonna use these tools, you do put it in YOLO mode and maybe hope for the best. Allow all tools and internet but lock down the path and use git if it starts deleting things [01:04:54] Scott: No. Well, if I'm standing in front of it and I'm actively doing it, usually I'll say like, allow all tools, because I know the tools that it has, I tend to lock them down by path. So for example, I'm in, d, GitHub, open live writer. [01:05:11] Scott: It can go ham on anything from there down. Um, it can talk to anything on the web and it can talk to anything from there on that path down. And it can call any tool. I'm sure it could decide randomly to format my hard drive, but it's not running as admin. So that would fail. Um, I suppose it could randomly have a reason to go out and, you know, hit the internet somehow and do something problematic. [01:05:36] Scott: But again, it's working on one specific thing. The loop that it's doing is looking at a log file, running a compiler and doing it. Again. That's different than Clawdbot. Or now called Moltbot which is a, basically Siri as a Ralph loop running on a, you know, a, a server somewhere or running in a, in a, in a background session. [01:05:59] Scott: But yes, you don't know. But, giving it access to like your source code and having, you know, if it deletes it, then I'll just go and git reset and I'll get it, you know, I'll get it back. I've had it do dumb stuff. I've had it, push when I didn't want it to push. I've had it commit stuff and then I just undo and go back to running. [01:06:18] Scott: I always run them in git. I wanna call that out though, like, I always have source control because if it makes a mistake, you back out. Agents make mistakes and even close themselves [01:06:26] Jeremy: Yeah, because I was thinking of the example you gave earlier where I believe it was looking at processes and killing 'em and things like that. So to, to me it seems like it must have quite a bit of permissions if it's able to do that. [01:06:40] Scott: Well, so when it runs as you, it has all the permissions that you have. So for example, the loop that I'm running over on this other machine, I, I told it that it can fire up the Windows app, save the PID of the app that it just ran. And it can kill that PID it can kill that process. [01:06:55] Scott: That doesn't mean it couldn't accidentally kill all of the processes. And I have one time, two weeks ago had, uh, Claude code kill itself by doing a stop process. And it was a little bit too, you know, *.*. And then it woke up and it's like, what happened? How long? It literally said how long have I been out [01:07:14] Jeremy: Wow. [01:07:15] Scott: Which I thought was crazy. Uh, it, it, it literally is like, how long have I been awake, you know? And I was like, just, and I said, check the markdown files, see where you left off and keep going. And it, you know, it picked it up, but it basically bonked itself on the head with its own hammer. Um, and, and that's not a guarantee. Telling the model what it can do is not a real sandbox, need to run in a container or isolate it somehow [01:07:32] Scott: So then the question is, is it a sandbox to put in a markdown file? Hey man, don't do that. No, it's not. That's obviously not a sandbox, so that's why running it in a container or you know, running it in WSL or whatever, you know, and that, that's non-specific of any operating system, if you decide to double click on a batch file and it does something stupid, I don't see that as being any different than running an agent. [01:07:55] Scott: The only difference is the ambiguity, which gets us back to the beginning of the conversation. Ambiguity loops are ambiguous. [01:08:03] Jeremy: Yeah, I mean, I could think of an example of if you're giving it access to the internet and maybe it downloads a utility, it think it's gonna help. [01:08:11] Scott: That, that's a [01:08:12] Jeremy: Yeah. And uh [01:08:14] Scott: Download a utility your company hasn't okayed and then suddenly you're in trouble. [01:08:17] Jeremy: Right. [01:08:18] Scott: Yep. Yeah, and I think that the difference between making a toy application that plays with your blood sugar in your spare time and working at a bank are different. So yeah, I would not go running anything like this, you know, in, in YOLO as admin in a bank [01:08:35] Jeremy: Makes sense. [01:08:35] Scott: That said. If it's your machine and you know what you're doing and you have this folder and down, there's a lot of power in this. [01:08:42] Scott: And the nice part about it is that it's a slider bar of good, better, best. You don't have to say it's not, it's not between bother me all the time and yolo. You can give it and you can do this as a team, like the whole enterprise could roll out, here are the five approved tools, here are the six approved websites, here's the SKILLS.md (http://SKILLS.md) that we're all gonna use. [01:09:00] Scott: We check it all in with the repo and everyone's on the same page. Differences between models [01:09:05] Jeremy: I think something you've alluded to during the conversation is you use different [01:09:10] Jeremy: agents and different models like there's Cloud Code or Gemini CLI or open I AI Codex. Are there specific strengths or or reasons that you use different agents or different models? [01:09:26] Scott: At this point? For me, as someone who wants to be an educated person that can go on a podcast with you and speak with some amount of. At least opinion, if not a little, tiny bit of authority. I try them all because I'm curious and I wanna understand them. For a lot of people it's just like Honda versus Toyota, like, which one did you like first? Uh, you'll have chat chats with people and be like, oh man, codex is amazing. Like, I get such great results with Codex. And then someone else think next to like, codex is trash and you there, it just depends on how you're talking to it. [01:09:59] Scott: Honda versus Toyota. I, I find that Opus is really good at planning. I find that Sonnet is really inexpensive and really good at executing on a plan. I find that Codex is very slow, but very good at, reviewing code. I don't know if that's true for everyone. I believe that it is true for me. Uh, I'm playing with, you know, Gemini and things like that, so I'm always open to trying new things, you know, like I went to Chipotle for a very long time. [01:10:28] Scott: I was very well known as a Chipotle guy. I have a shirt on right now that literally says, will code for tacos, but now I'm going to Qdoba more and more and more. So, strong opinions weekly held should be your perspective on models and like, I'm here, I just said /model in copilot CLI, and there's 14 models to choose from. [01:10:48] Scott: You can turn them on and off based on what your organization has decided. So I see a lot of models because I have an individual subscription. You might log in and see four, and then you have to go in and turn on the ones you want for your enterprise, decide how much you wanna pay, and then your company might decide, or your CTO might say, oh, we're, we're A GPT shop. [01:11:09] Scott: Oh, we're a cla a Claude shop. Well, with something like Claude Code, you get Claude models with some of the things like OpenCode or amp. You get an auto model, which is another interesting thing by the way. Some people, like you don't actually, when you order an Uber, say I want a Honda, you get whatever comes. [01:11:26] Scott: You might say Uber black, or you might say whatever, but you don't really care that. I think we're gonna start seeing that more and more where it doesn't matter what model, you're gonna have an auto router and it won't say Claude Opus or GPT, it'll say auto, or it'll say Smart, or it'll say fast, or it'll say deep and then we'll do the right thing and you're not gonna worry about it as much. Statically typed languages better for LLMs? [01:11:48] Jeremy: I had an interview with Bryan cantrill he's the CTO of Oxide computer. [01:11:55] Scott: Yeah. Yeah. I was on ox oxide and friends. [01:11:57] Jeremy: Oh, nice. And one of the things he, he mentioned with working with large language models is he thought it was really advantageous to use strongly type languages like say rust, because the model is able to verify when things are not correct, just through that compile step and through the design of the language. [01:12:23] Jeremy: I wonder in your impression, like whether you found that true for yourself. Like are there specific languages that work better when using these agents? [01:12:35] Scott: That is a very good question. I think that is a good opinion by Bryan and as a general rule, I agree with it, but I would argue that in lieu of strong typing, one overcompensates with tests. So correctness is what Bryan is arguing for and that is valid. If I am using Python like I am for my blood sugar thing, having 200 tests make me worry less about how Python compiles or doesn't. With my .NET application, I have fewer tests, but I've got 419, I'm looking at this 419,000 lines of code and it all compiles that. But compilation doesn't equal correctness and strong typing doesn't guarantee correctness. So I would say I mostly agree with Bryan, but if you go back to fear-driven development, if you're afraid of the code, you don't have enough tests. So I write tests until I'm no longer afraid, regardless of language. I would say I am picking the right language for the environment, not picking a language based on whether it's strongly typed or not. [01:13:48] Jeremy: Probably a similar philosophy to when you're writing the code yourself, even [01:13:53] Jeremy: before these tools. Writing more code in languages you don't know but fit the domain [01:13:54] Scott: A hundred percent. Although I am finding myself writing more code in languages that I don't know because it's the right language. Like not everything has to be a, a, a .NET application that's like a complete self-contained cross-platform thing. Sometimes a hundred lines of Python is fine. [01:14:16] Jeremy: Yeah. I think that would be an interesting shift for people because I, I think there does tend to be people who they know, let's say they know JavaScript, so they go, okay, I'm gonna write everything in JavaScript or Node or TypeScript and maybe this this removes some of the, the difficulty in getting started. [01:14:38] Scott: [01:14:38] Jeremy: Let's say Python as an example, if you're working on something in the machine learning space, [01:14:43] Scott: Yep. [01:14:44] Jeremy: Even if you're not an expert in Python, these tools can, [01:14:47] Scott: [01:14:47] Jeremy: help get you there. [01:14:49] Scott: Well, like the, the, the, the, the blood sugar thing is meant to be a skill that is plugged into an agent. So you put it in, see users, Jeremy copilot skills or see users jeremy Claude skills. So I did it in Python because it's like one file, one markdown file, and one Python file. And you have Python. And I have Python. [01:15:07] Scott: It works. I could have made it a .NET application. It would be larger, it would require a runtime. So that wasn't necessary. So I used Python for portability. Even though I don't know Python, I'm confident that it will work for you and it'll have a good experience. But you know, like I might have, I'm trying to think which one of these is a net application. [01:15:25] Scott: like my, my live writer application, I wouldn't do it in anything but .NET 'cause that's what it's good at. So pick, pick the car brand that is gonna make your trip the most delightful. I'm definitely focused on delight right now. [01:15:40] Jeremy: I think that's pretty good place to end it on, but is there anything else that you think we should have talked about or you wanna mention? [01:15:48] Scott: I just think that it's a funny moment that we're in where people who are craftspeople are worried that the craft will go away. And I hope that it is just that the toil will go away, but the craft remains. I am having a blast. I hope I will be having a blast next this time next year. Maybe we'll chat and see how it turned out. Difference in how people think of the tools [01:16:11] Jeremy: Yeah, I think it's so interesting how different people are reacting to these tools. You have some people where it seems like. This is amazing. I'm doing all these things, and then you have some people where it's like, I spend more time just trying to fix all the problems and this is not that helpful to me. [01:16:32] Jeremy: And so there's this big disconnect I feel like, between how people think of these tools. [01:16:38] Scott: Yeah, I think that's okay. Like if you love driving Stick and you have BMWs and you just, the, the joy of driving then don't take an Uber, right? But they're, they're solving different problems. Taking an Uber pick someone up who maybe can't drive, it enables them to do something that they could not do before. [01:17:01] Scott: While driving stick shift around a track for fun at speed is different. So, yeah, I mean, I'm writing, like, I'm writing software for this robot over here. I'm writing software for my Commodore 64. and I'm writing software for Windows and Linux and Mac and, I'm using different tools in different languages in each each one. [01:17:20] Scott: Some are more AI augmented because it matters more or less how much of myself I pour into the thing. But I feel like the result is so far quality software that helps people. [01:17:35] Jeremy: Yeah, I think there's this mix of, for some people it's the effectiveness of the tool for them personally. And then I think there's the other part, like you're, you're describing the, the stick shift example. And I think what people kind of wonder is right now you can still get a car with a stick shift and, and drive it. [01:17:58] Jeremy: And going into the future, if we are a person who wants to manually write the code, we want to craft it in a very specific way. Are we going to commercially still be able to continue to allow to be, do that. [01:18:16] Scott: That is more of an economics problem than a tools problem. I subscribe to Trek economics, uh, which is the science and, uh, economics of Star Trek. It's an excellent book and worth looking up. And the whole point of Trek omics is that, craftsmanship matters. And just because you can say Tea Earl Gray Hot and it pops out of a door, that doesn't mean that you shouldn't make yourself a nice cup of tea sometimes on your own. [01:18:45] Jeremy: All right. you mentioned you have a podcast, you actually have many podcasts. Um, if people wanna check out what you're up to, uh, where should they head? [01:18:53] Scott: Go to hanselman.com (http://hanselman.com), hanselman.com (http://hanselman.com), my last name. If there is a tariff on podcasts, I'm afraid I will have to pay the tariff. [01:19:00] Scott: Uh, I tend to yap and I apologize for that. But, Hanselminutes is my podcast you can get to at hanselman.com (http://hanselman.com). It's been around over a thousand episodes over the last 20 years. And then I have a new podcast with Mark Russinovich, the CTO of Azure, where we learn something new every week. It's called Mark and Scott Learn to. [01:19:15] Scott: So those are two that people can check out. [01:19:18] Jeremy: Yeah, I highly recommend both of those. It's also amazing how long you've been doing Hanselminutes. I think it's been, you said over 20 years Right. [01:19:26] Scott: Yeah, it's been a minute. It's been a minute thou every Thursday for the last thousand plus episodes. And it, it, it, it tries to showcase faces that you maybe not ordinarily see. [01:19:35] Scott: There's like the same three or four people that do the shows and, uh, I'm, I'm guilty of that, but there's a lot of cool people doing a lot of amazing work. I recently had, uh, a lady who is the IT manager of the Philadelphia Airport and I learned all about how airports run and like you don't see folks like that on a podcast. [01:19:52] Scott: That's amazing. So yeah, definitely showcasing of people that are doing great work. [01:19:57] Jeremy: All right. [01:19:58] Jeremy: We're good. [01:20:01] Scott: I thought that was fun. [01:20:02] Jeremy: Yeah, for sure. Me too. [01:20:04] Scott: See, if Grant tells me I broke into jail or anything, Grant's like, shut up. No, we don't talk about pricing. [01:20:11] Jeremy: You didn't have a chaperone last time. Things have changed. [01:20:14] Scott: I don't know man. Dude you become important at work, it's like a whole thing. I accidentally became important at work. [01:20:21] Scott: Scott, you are, you are free and clear. No jail for you. I, I mean, I had a great time listening, so, uh, I'm glad that I did. Cool. These tools give new developers confidence to try things [01:20:29] Scott: Well I know that we're still recording. You can use this as B roll if you want. [01:20:33] Scott: But like, I was doing this talk internally recently for, the developer relations people and they're like, you know, what are we selling here man? What are we even selling, bro? Are we selling like tokens? Like we're trying to get people to use copilot. Like what's the thing? It's confidence transfer. I am confident that I could make whatever the (blank) I want right now. [01:20:51] Scott: And it's amazing. I am. Just building stuff, and it's like, it's delightful. I feel like my 30 years of software is coming up to this moment where I can now make anything. But if a young person, an early career, early in career person, a new builder, a fresh builder, someone who just got outta school, just, just busted their ass for four years and just got a computer science degree, and they don't feel like they have that confidence, and they're like, oh, man, my job is over. [01:21:18] Scott: I want to transfer my confidence to them. I want them to feel like, no, this is perfect. You entered at the perfect time. You know, computer science, you have a degree in computer science. You learn how to do this stuff manually. You know how to drive stick shift, and now we're gonna give you an Uber. You have the best of both worlds. [01:21:36] Scott: And I want them to feel like that. And if, if, if a, if a copilot or a Claude makes someone feel dumb, we're doing it wrong. We have to let people know that this is a exoskeleton for your brain. But I don't want people to end up like the end of Wall-E where it's just like. In a chair and they're just bringing tacos to you, you know, via Amazon Prime. [01:21:56] Scott: You've gotta do the work, you gotta know how the thing works, and if you do, you're gonna have a blast. [01:22:02] Jeremy: [01:22:02] Scott: It does tickle a part of your mind, because I talked about taste. Like how do you know when it sucks or not? You just, I don't know. I mean, I just, I just know, bro. [01:22:12] Scott: You know what I mean? And, and that feeling of like, yeah, I know that, that, that feels right or that doesn't feel right. Developing that. I'll tell you one more story and then I'll get outta your way. Okay, so we're doing this new thing called a preceptorship. Most people have like internships and apprenticeships, and that makes the, the person who is less than they have a label, right? [01:22:35] Scott: Jeremy, you're an apprentice. Work really hard and pull yourself out of apprentice world. My wife is a nurse and she is a nurse preceptor. A preceptor is a nurse trainer. They, she has an actual name tag that says preceptor. So if an, if a junior nurse says, Hey, um, I have a question, she's a safe person. [01:22:56] Scott: They're not gonna like, flip the bozo bit and like fire her. Like, if an apprentice or an intern says something dumb in a meeting, they're gonna like, oh my God, I'm gonna lose my job. But if a nurse preceptor is there, they're a safe person to ask questions for. And, um, do you watch The Pitt? You know, No, no Oh, dude, so good. Okay. But you've seen like, er, you've seen like health shows, stuff like that. Okay. So. On the pitt. Like there's always a moment or any health movie like this where like some guy comes in and he is choking and he can't breathe and he's like, and somebody says, oh my god, quick Jeremy, do you have a pen? [01:23:30] Scott: And then Jeremy hands me the pen and I pull the pen parts out and I have like a tube and I'm like, alright, I gotta stab this guy in the neck. You shove the neck and stab the guy in the neck and then he can breathe in his airway and he goes, I can breathe. Right. What they do on these shows is that they'll say, Jeremy, have you ever stabbed a guy in the neck with a pen? [01:23:48] Scott: And you're like, no dude, it's my first day. I just got here. I'm a junior nurse, dude, this is your chance. Gives you the thing, you're like, Hey, you gotta put it by the third and fourth thoracic, whatever. And you go. And then at the end when the guy's alive and there's blood everywhere and the pen is like sticking out of his neck and they wheel him away, they say, Jeremy, how is that? [01:24:06] Scott: That was amazing. I'm gonna be a doctor. I want that for freaking software engineers. Right sites down. Like that's how I grew up. Pair programming sites down. Oh man, Jeremy broke production. What are we gonna do? Time to learn everybody quick, grab a pen. You ever stabbed a whip server? I don't know the metaphor falls apart at this point, but you know, let's go and stab servers in the neck with a pen and learn and get excited and say, yeah, I can make stuff. [01:24:37] Jeremy: That makes sense to me. I mean, like I, I remember when I was like tutoring in computer labs for some people, when they had a problem, what they really needed was the start to be kind of filled in. [01:24:50] Scott: Yep, exactly. [01:24:52] Jeremy: And I feel like a lot of these tools, if you're not sure how to get started or you're kind of scared of getting started, they can at least get you something. [01:25:01] Jeremy: And then based on that, you're like, okay. Like I kinda, I kinda know how I can start filling in the rest or how I can start moving forward. Yeah. [01:25:07] Scott: Exactly. Alright. Mac mini person is texting me. On Facebook [01:25:12] Scott: Marketplace. They've left it on the porch so I gotta go.

3. 

   27 Feb 2026

   Bryan Cantrill on Oxide Computer

   Bryan Cantrill is the co-founder and CTO of Oxide Computer Company. We discuss why the biggest cloud providers don't use off the shelf hardware, how scaling data centers at samsung's scale exposed problems with hard drive firmware, how the values of NodeJS are in conflict with robust systems, choosing Rust, and the benefits of Oxide Computer's rack scale approach. This is an extended version of an interview posted on Software Engineering Radio. Related links Oxide Computer (https://oxide.computer) Oxide and Friends (https://oxide-and-friends.transistor.fm) Illumos (https://illumos.org) Platform as a Reflection of Values (https://www.youtube.com/watch?v=Xhx970_JKX4) RFD 26 (https://rfd.shared.oxide.computer/rfd/26) bhyve (https://bhyve.org) CockroachDB (https://www.cockroachlabs.com) Heterogeneous Computing with Raja Koduri (https://oxide-and-friends.transistor.fm/episodes/heterogeneous-computing-with-raja-koduri) Transcript You can help correct transcripts on GitHub (https://github.com/jeremyjung/softwaresessions/tree/master/content/episodes). Intro [00:00:00] Jeremy: Today I am talking to Bryan Cantrill. He's the co-founder and CTO of Oxide computer company, and he was previously the CTO of Joyent and he also co-authored the DTrace Tracing framework while he was at Sun Microsystems. [00:00:14] Jeremy: Bryan, welcome to Software Engineering radio. [00:00:17] Bryan: Uh, awesome. Thanks for having me. It's great to be here. [00:00:20] Jeremy: You're the CTO of a company that makes computers. But I think before we get into that, a lot of people who built software, now that the actual computer is abstracted away, they're using AWS or they're using some kind of cloud service. So I thought we could start by talking about, data centers. [00:00:41] Jeremy: 'cause you were. Previously working at Joyent, and I believe you got bought by Samsung and you've previously talked about how you had to figure out, how do I run things at Samsung's scale. So how, how, how was your experience with that? What, what were the challenges there? Samsung scale and migrating off the cloud [00:01:01] Bryan: Yeah, I mean, so at Joyent, and so Joyent was a cloud computing pioneer. Uh, we competed with the likes of AWS and then later GCP and Azure. Uh, and we, I mean, we were operating at a scale, right? We had a bunch of machines, a bunch of dcs, but ultimately we know we were a VC backed company and, you know, a small company by the standards of, certainly by Samsung standards. [00:01:25] Bryan: And so when, when Samsung bought the company, I mean, the reason by the way that Samsung bought Joyent is Samsung's. Cloud Bill was, uh, let's just say it was extremely large. They were spending an enormous amount of money every year on, on the public cloud. And they realized that in order to secure their fate economically, they had to be running on their own infrastructure. [00:01:51] Bryan: It did not make sense. And there's not, was not really a product that Samsung could go buy that would give them that on-prem cloud. Uh, I mean in that, in that regard, like the state of the market was really no different. And so they went looking for a company, uh, and bought, bought Joyent. And when we were on the inside of Samsung. [00:02:11] Bryan: That we learned about Samsung scale. And Samsung loves to talk about Samsung scale. And I gotta tell you, it is more than just chest thumping. Like Samsung Scale really is, I mean, just the, the sheer, the number of devices, the number of customers, just this absolute size. they really wanted to take us out to, to levels of scale, certainly that we had not seen. [00:02:31] Bryan: The reason for buying Joyent was to be able to stand up on their own infrastructure so that we were gonna go buy, we did go buy a bunch of hardware. Problems with server hardware at scale [00:02:40] Bryan: And I remember just thinking, God, I hope Dell is somehow magically better. I hope the problems that we have seen in the small, we just. You know, I just remember hoping and hope is hope. It was of course, a terrible strategy and it was a terrible strategy here too. Uh, and the we that the problems that we saw at the large were, and when you scale out the problems that you see kind of once or twice, you now see all the time and they become absolutely debilitating. [00:03:12] Bryan: And we saw a whole series of really debilitating problems. I mean, many ways, like comically debilitating, uh, in terms of, of showing just how bad the state-of-the-art. Yes. And we had, I mean, it should be said, we had great software and great software expertise, um, and we were controlling our own system software. [00:03:35] Bryan: But even controlling your own system software, your own host OS, your own control plane, which is what we had at Joyent, ultimately, you're pretty limited. You go, I mean, you got the problems that you can obviously solve, the ones that are in your own software, but the problems that are beneath you, the, the problems that are in the hardware platform, the problems that are in the componentry beneath you become the problems that are in the firmware. IO latency due to hard drive firmware [00:04:00] Bryan: Those problems become unresolvable and they are deeply, deeply frustrating. Um, and we just saw a bunch of 'em again, they were. Comical in retrospect, and I'll give you like a, a couple of concrete examples just to give, give you an idea of what kinda what you're looking at. one of the, our data centers had really pathological IO latency. [00:04:23] Bryan: we had a very, uh, database heavy workload. And this was kind of right at the period where you were still deploying on rotating media on hard drives. So this is like, so. An all flash buy did not make economic sense when we did this in, in 2016. This probably, it'd be interesting to know like when was the, the kind of the last time that that actual hard drives made sense? [00:04:50] Bryan: 'cause I feel this was close to it. So we had a, a bunch of, of a pathological IO problems, but we had one data center in which the outliers were actually quite a bit worse and there was so much going on in that system. It took us a long time to figure out like why. And because when, when you, when you're io when you're seeing worse io I mean you're naturally, you wanna understand like what's the workload doing? [00:05:14] Bryan: You're trying to take a first principles approach. What's the workload doing? So this is a very intensive database workload to support the, the object storage system that we had built called Manta. And that the, the metadata tier was stored and uh, was we were using Postgres for that. And that was just getting absolutely slaughtered. [00:05:34] Bryan: Um, and ultimately very IO bound with these kind of pathological IO latencies. Uh, and as we, you know, trying to like peel away the layers to figure out what was going on. And I finally had this thing. So it's like, okay, we are seeing at the, at the device layer, at the at, at the disc layer, we are seeing pathological outliers in this data center that we're not seeing anywhere else. [00:06:00] Bryan: And that does not make any sense. And the thought occurred to me. I'm like, well, maybe we are. Do we have like different. Different rev of firmware on our HGST drives, HGST. Now part of WD Western Digital were the drives that we had everywhere. And, um, so maybe we had a different, maybe I had a firmware bug. [00:06:20] Bryan: I, this would not be the first time in my life at all that I would have a drive firmware issue. Uh, and I went to go pull the firmware, rev, and I'm like, Toshiba makes hard drives? So we had, I mean. I had no idea that Toshiba even made hard drives, let alone that they were our, they were in our data center. [00:06:38] Bryan: I'm like, what is this? And as it turns out, and this is, you know, part of the, the challenge when you don't have an integrated system, which not to pick on them, but Dell doesn't, and what Dell would routinely put just sub make substitutes, and they make substitutes that they, you know, it's kind of like you're going to like, I don't know, Instacart or whatever, and they're out of the thing that you want. [00:07:03] Bryan: So, you know, you're, someone makes a substitute and like sometimes that's okay, but it's really not okay in a data center. And you really want to develop and validate a, an end-to-end integrated system. And in this case, like Toshiba doesn't, I mean, Toshiba does make hard drives, but they are a, or the data they did, uh, they basically were, uh, not competitive and they were not competitive in part for the reasons that we were discovering. [00:07:29] Bryan: They had really serious firmware issues. So the, these were drives that would just simply stop a, a stop acknowledging any reads from the order of 2,700 milliseconds. Long time, 2.7 seconds. Um. And that was a, it was a drive firmware issue, but it was highlighted like a much deeper issue, which was the simple lack of control that we had over our own destiny. [00:07:53] Bryan: Um, and it's an, it's, it's an example among many where Dell is making a decision. That lowers the cost of what they are providing you marginally, but it is then giving you a system that they shouldn't have any confidence in because it's not one that they've actually designed and they leave it to the customer, the end user, to make these discoveries. [00:08:18] Bryan: And these things happen up and down the stack. And for every, for whether it's, and, and not just to pick on Dell because it's, it's true for HPE, it's true for super micro, uh, it's true for your switch vendors. It's, it's true for storage vendors where the, the, the, the one that is left actually integrating these things and trying to make the the whole thing work is the end user sitting in their data center. AWS / Google are not buying off the shelf hardware but you can't use it [00:08:42] Bryan: There's not a product that they can buy that gives them elastic infrastructure, a cloud in their own DC The, the product that you buy is the public cloud. Like when you go in the public cloud, you don't worry about the stuff because that it's, it's AWS's issue or it's GCP's issue. And they are the ones that get this to ground. [00:09:02] Bryan: And they, and this was kind of, you know, the eye-opening moment. Not a surprise. Uh, they are not Dell customers. They're not HPE customers. They're not super micro customers. They have designed their own machines. And to varying degrees, depending on which one you're looking at. But they've taken the clean sheet of paper and the frustration that we had kind of at Joyent and beginning to wonder and then Samsung and kind of wondering what was next, uh, is that, that what they built was not available for purchase in the data center. [00:09:35] Bryan: You could only rent it in the public cloud. And our big belief is that public cloud computing is a really important revolution in infrastructure. Doesn't feel like a different, a deep thought, but cloud computing is a really important revolution. It shouldn't only be available to rent. You should be able to actually buy it. [00:09:53] Bryan: And there are a bunch of reasons for doing that. Uh, one in the one we we saw at Samsung is economics, which I think is still the dominant reason where it just does not make sense to rent all of your compute in perpetuity. But there are other reasons too. There's security, there's risk management, there's latency. [00:10:07] Bryan: There are a bunch of reasons why one might wanna to own one's own infrastructure. But, uh, that was very much the, the, so the, the genesis for oxide was coming out of this very painful experience and a painful experience that, because, I mean, a long answer to your question about like what was it like to be at Samsung scale? [00:10:27] Bryan: Those are the kinds of things that we, I mean, in our other data centers, we didn't have Toshiba drives. We only had the HDSC drives, but it's only when you get to this larger scale that you begin to see some of these pathologies. But these pathologies then are really debilitating in terms of those who are trying to develop a service on top of them. [00:10:45] Bryan: So it was, it was very educational in, in that regard. And you're very grateful for the experience at Samsung in terms of opening our eyes to the challenge of running at that kind of scale. [00:10:57] Jeremy: Yeah, because I, I think as software engineers, a lot of times we, we treat the hardware as a, as a given where, [00:11:08] Bryan: Yeah. [00:11:08] Bryan: Yeah. There's software in chard drives [00:11:09] Jeremy: It sounds like in, in this case, I mean, maybe the issue is not so much that. Dell or HP as a company doesn't own every single piece that they're providing you, but rather the fact that they're swapping pieces in and out without advertising them, and then when it becomes a problem, they're not necessarily willing to, to deal with the, the consequences of that. [00:11:34] Bryan: They just don't know. I mean, I think they just genuinely don't know. I mean, I think that they, it's not like they're making a deliberate decision to kind of ship garbage. It's just that they are making, I mean, I think it's exactly what you said about like, not thinking about the hardware. It's like, what's a hard drive? [00:11:47] Bryan: Like what's it, I mean, it's a hard drive. It's got the same specs as this other hard drive and Intel. You know, it's a little bit cheaper, so why not? It's like, well, like there's some reasons why not, and one of the reasons why not is like, uh, even a hard drive, whether it's rotating media or, or flash, like that's not just hardware. [00:12:05] Bryan: There's software in there. And that the software's like not the same. I mean, there are components where it's like, there's actually, whether, you know, if, if you're looking at like a resistor or a capacitor or something like this Yeah. If you've got two, two parts that are within the same tolerance. Yeah. [00:12:19] Bryan: Like sure. Maybe, although even the EEs I think would be, would be, uh, objecting that a little bit. But the, the, the more complicated you get, and certainly once you get to the, the, the, the kind of the hardware that we think of like a, a, a microprocessor, a a network interface card, a a, a hard driver, an NVME drive. [00:12:38] Bryan: Those things are super complicated and there's a whole bunch of software inside of those things, the firmware, and that's the stuff that, that you can't, I mean, you say that software engineers don't think about that. It's like you, no one can really think about that because it's proprietary that's kinda welded shut and you've got this abstraction into it. [00:12:55] Bryan: But the, the way that thing operates is very core to how the thing in aggregate will behave. And I think that you, the, the kind of, the, the fundamental difference between Oxide's approach and the approach that you get at a Dell HP Supermicro, wherever, is really thinking holistically in terms of hardware and software together in a system that, that ultimately delivers cloud computing to a user. [00:13:22] Bryan: And there's a lot of software at many, many, many, many different layers. And it's very important to think about, about that software and that hardware holistically as a single system. [00:13:34] Jeremy: And during that time at Joyent, when you experienced some of these issues, was it more of a case of you didn't have enough servers experiencing this? So if it would happen, you might say like, well, this one's not working, so maybe we'll just replace the hardware. What, what was the thought process when you were working at that smaller scale and, and how did these issues affect you? UEFI / Baseboard Management Controller [00:13:58] Bryan: Yeah, at the smaller scale, you, uh, you see fewer of them, right? You just see it's like, okay, we, you know, what you might see is like, that's weird. We kinda saw this in one machine versus seeing it in a hundred or a thousand or 10,000. Um, so you just, you just see them, uh, less frequently as a result, they are less debilitating. [00:14:16] Bryan: Um, I, I think that it's, when you go to that larger scale, those things that become, that were unusual now become routine and they become debilitating. Um, so it, it really is in many regards a function of scale. Uh, and then I think it was also, you know, it was a little bit dispiriting that kind of the substrate we were building on really had not improved. [00:14:39] Bryan: Um, and if you look at, you know, the, if you buy a computer server, buy an x86 server. There is a very low layer of firmware, the BIOS, the basic input output system, the UEFI BIOS, and this is like an abstraction layer that has, has existed since the eighties and hasn't really meaningfully improved. Um, the, the kind of the transition to UEFI happened with, I mean, I, I ironically with Itanium, um, you know, two decades ago. [00:15:08] Bryan: but beyond that, like this low layer, this lowest layer of platform enablement software is really only impeding the operability of the system. Um, you look at the baseboard management controller, which is the kind of the computer within the computer, there is a, uh, there is an element in the machine that needs to handle environmentals, that needs to handle, uh, operate the fans and so on. [00:15:31] Bryan: Uh, and that traditionally has this, the space board management controller, and that architecturally just hasn't improved in the last two decades. And, you know, that's, it's a proprietary piece of silicon. Generally from a company that no one's ever heard of called a Speed, uh, which has to be, is written all on caps, so I guess it needs to be screamed. [00:15:50] Bryan: Um, a speed has a proprietary part that has a, there is a root password infamously there, is there, the root password is encoded effectively in silicon. So, uh, which is just, and for, um, anyone who kind of goes deep into these things, like, oh my God, are you kidding me? Um, when we first started oxide, the wifi password was a fraction of the a speed root password for the bmc. [00:16:16] Bryan: It's kinda like a little, little BMC humor. Um, but those things, it was just dispiriting that, that the, the state-of-the-art was still basically personal computers running in the data center. Um, and that's part of what, what was the motivation for doing something new? [00:16:32] Jeremy: And for the people using these systems, whether it's the baseboard management controller or it's the The BIOS or UF UEFI component, what are the actual problems that people are seeing seen? Security vulnerabilities and poor practices in the BMC [00:16:51] Bryan: Oh man, I, the, you are going to have like some fraction of your listeners, maybe a big fraction where like, yeah, like what are the problems? That's a good question. And then you're gonna have the people that actually deal with these things who are, did like their heads already hit the desk being like, what are the problems? [00:17:06] Bryan: Like what are the non problems? Like what, what works? Actually, that's like a shorter answer. Um, I mean, there are so many problems and a lot of it is just like, I mean, there are problems just architecturally these things are just so, I mean, and you could, they're the problems spread to the horizon, so you can kind of start wherever you want. [00:17:24] Bryan: But I mean, as like, as a really concrete example. Okay, so the, the BMCs that, that the computer within the computer that needs to be on its own network. So you now have like not one network, you got two networks that, and that network, by the way, it, that's the network that you're gonna log into to like reset the machine when it's otherwise unresponsive. [00:17:44] Bryan: So that going into the BMC, you can are, you're able to control the entire machine. Well it's like, alright, so now I've got a second net network that I need to manage. What is running on the BMC? Well, it's running some. Ancient, ancient version of Linux it that you got. It's like, well how do I, how do I patch that? [00:18:02] Bryan: How do I like manage the vulnerabilities with that? Because if someone is able to root your BMC, they control the system. So it's like, this is not you've, and now you've gotta go deal with all of the operational hair around that. How do you upgrade that system updating the BMC? I mean, it's like you've got this like second shadow bad infrastructure that you have to go manage. [00:18:23] Bryan: Generally not open source. There's something called open BMC, um, which, um, you people use to varying degrees, but you're generally stuck with the proprietary BMC, so you're generally stuck with, with iLO from HPE or iDRAC from Dell or, or, uh, the, uh, su super micros, BMC, that H-P-B-M-C, and you are, uh, it is just excruciating pain. [00:18:49] Bryan: Um, and that this is assuming that by the way, that everything is behaving correctly. The, the problem is that these things often don't behave correctly, and then the consequence of them not behaving correctly. It's really dire because it's at that lowest layer of the system. So, I mean, I'll give you a concrete example. [00:19:07] Bryan: a customer of theirs reported to me, so I won't disclose the vendor, but let's just say that a well-known vendor had an issue with their, their temperature sensors were broken. Um, and the thing would always read basically the wrong value. So it was the BMC that had to like, invent its own ki a different kind of thermal control loop. [00:19:28] Bryan: And it would index on the, on the, the, the, the actual inrush current. It would, they would look at that at the current that's going into the CPU to adjust the fan speed. That's a great example of something like that's a, that's an interesting idea. That doesn't work. 'cause that's actually not the temperature. [00:19:45] Bryan: So like that software would crank the fans whenever you had an inrush of current and this customer had a workload that would spike the current and by it, when it would spike the current, the, the, the fans would kick up and then they would slowly degrade over time. Well, this workload was spiking the current faster than the fans would degrade, but not fast enough to actually heat up the part. [00:20:08] Bryan: And ultimately over a very long time, in a very painful investigation, it's customer determined that like my fans are cranked in my data center for no reason. We're blowing cold air. And it's like that, this is on the order of like a hundred watts, a server of, of energy that you shouldn't be spending and like that ultimately what that go comes down to this kind of broken software hardware interface at the lowest layer that has real meaningful consequence, uh, in terms of hundreds of kilowatts, um, across a data center. So this stuff has, has very, very, very real consequence and it's such a shadowy world. Part of the reason that, that your listeners that have dealt with this, that our heads will hit the desk is because it is really aggravating to deal with problems with this layer. [00:21:01] Bryan: You, you feel powerless. You don't control or really see the software that's on them. It's generally proprietary. You are relying on your vendor. Your vendor is telling you that like, boy, I don't know. You're the only customer seeing this. I mean, the number of times I have heard that for, and I, I have pledged that we're, we're not gonna say that at oxide because it's such an unaskable thing to say like, you're the only customer saying this. [00:21:25] Bryan: It's like, it feels like, are you blaming me for my problem? Feels like you're blaming me for my problem? Um, and what you begin to realize is that to a degree, these folks are speaking their own truth because the, the folks that are running at real scale at Hyperscale, those folks aren't Dell, HP super micro customers. [00:21:46] Bryan: They're actually, they've done their own thing. So it's like, yeah, Dell's not seeing that problem, um, because they're not running at the same scale. Um, but when you do run, you only have to run at modest scale before these things just become. Overwhelming in terms of the, the headwind that they present to people that wanna deploy infrastructure. The problem is felt with just a few racks [00:22:05] Jeremy: Yeah, so maybe to help people get some perspective at, at what point do you think that people start noticing or start feeling these problems? Because I imagine that if you're just have a few racks or [00:22:22] Bryan: do you have a couple racks or the, or do you wonder or just wondering because No, no, no. I would think, I think anyone who deploys any number of servers, especially now, especially if your experience is only in the cloud, you're gonna be like, what the hell is this? I mean, just again, just to get this thing working at all. [00:22:39] Bryan: It is so it, it's so hairy and so congealed, right? It's not designed. Um, and it, it, it, it's accreted it and it's so obviously accreted that you are, I mean, nobody who is setting up a rack of servers is gonna think to themselves like, yes, this is the right way to go do it. This all makes sense because it's, it's just not, it, I, it feels like the kit, I mean, kit car's almost too generous because it implies that there's like a set of plans to work to in the end. [00:23:08] Bryan: Uh, I mean, it, it, it's a bag of bolts. It's a bunch of parts that you're putting together. And so even at the smallest scales, that stuff is painful. Just architecturally, it's painful at the small scale then, but at least you can get it working. I think the stuff that then becomes debilitating at larger scale are the things that are, are worse than just like, I can't, like this thing is a mess to get working. [00:23:31] Bryan: It's like the, the, the fan issue that, um, where you are now seeing this over, you know, hundreds of machines or thousands of machines. Um, so I, it is painful at more or less all levels of scale. There's, there is no level at which the, the, the pc, which is really what this is, this is a, the, the personal computer architecture from the 1980s and there is really no level of scale where that's the right unit. Running elastic infrastructure is the hardware but also, hypervisor, distributed database, api, etc [00:23:57] Bryan: I mean, where that's the right thing to go deploy, especially if what you are trying to run. Is elastic infrastructure, a cloud. Because the other thing is like we, we've kinda been talking a lot about that hardware layer. Like hardware is, is just the start. Like you actually gotta go put software on that and actually run that as elastic infrastructure. [00:24:16] Bryan: So you need a hypervisor. Yes. But you need a lot more than that. You, you need to actually, you, you need a distributed database, you need web endpoints. You need, you need a CLI, you need all the stuff that you need to actually go run an actual service of compute or networking or storage. I mean, and for, for compute, even for compute, there's a ton of work to be done. [00:24:39] Bryan: And compute is by far, I would say the simplest of the, of the three. When you look at like networks, network services, storage services, there's a whole bunch of stuff that you need to go build in terms of distributed systems to actually offer that as a cloud. So it, I mean, it is painful at more or less every LE level if you are trying to deploy cloud computing on. What's a control plane? [00:25:00] Jeremy: And for someone who doesn't have experience building or working with this type of infrastructure, when you talk about a control plane, what, what does that do in the context of this system? [00:25:16] Bryan: So control plane is the thing that is, that is everything between your API request and that infrastructure actually being acted upon. So you go say, Hey, I, I want a provision, a vm. Okay, great. We've got a whole bunch of things we're gonna provision with that. We're gonna provision a vm, we're gonna get some storage that's gonna go along with that, that's got a network storage service that's gonna come out of, uh, we've got a virtual network that we're gonna either create or attach to. [00:25:39] Bryan: We've got a, a whole bunch of things we need to go do for that. For all of these things, there are metadata components that need, we need to keep track of this thing that, beyond the actual infrastructure that we create. And then we need to go actually, like act on the actual compute elements, the hostos, what have you, the switches, what have you, and actually go. [00:25:56] Bryan: Create these underlying things and then connect them. And there's of course, the challenge of just getting that working is a big challenge. Um, but getting that working robustly, getting that working is, you know, when you go to provision of vm, um, the, all the, the, the steps that need to happen and what happens if one of those steps fails along the way? [00:26:17] Bryan: What happens if, you know, one thing we're very mindful of is these kind of, you get these long tails of like, why, you know, generally our VM provisioning happened within this time, but we get these long tails where it takes much longer. What's going on? What, where in this process are we, are we actually spending time? [00:26:33] Bryan: Uh, and there's a whole lot of complexity that you need to go deal with that. There's a lot of complexity that you need to go deal with this effectively, this workflow that's gonna go create these things and manage them. Um, we use a, a pattern that we call, that are called sagas, actually is a, is a database pattern from the eighties. [00:26:51] Bryan: Uh, Katie McCaffrey is a, is a database reCrcher who, who, uh, I, I think, uh, reintroduce the idea of, of sagas, um, in the last kind of decade. Um, and this is something that we picked up, um, and I've done a lot of really interesting things with, um, to allow for, to this kind of, these workflows to be, to be managed and done so robustly in a way that you can restart them and so on. [00:27:16] Bryan: Uh, and then you guys, you get this whole distributed system that can do all this. That whole distributed system, that itself needs to be reliable and available. So if you, you know, you need to be able to, what happens if you, if you pull a sled or if a sled fails, how does the system deal with that? [00:27:33] Bryan: How does the system deal with getting an another sled added to the system? Like how do you actually grow this distributed system? And then how do you update it? How do you actually go from one version to the next? And all of that has to happen across an air gap where this is gonna run as part of the computer. [00:27:49] Bryan: So there are, it, it is fractally complicated. There, there is a lot of complexity here in, in software, in the software system and all of that. We kind of, we call the control plane. Um, and it, this is the what exists at AWS at GCP, at Azure. When you are hitting an endpoint that's provisioning an EC2 instance for you. [00:28:10] Bryan: There is an AWS control plane that is, is doing all of this and has, uh, some of these similar aspects and certainly some of these similar challenges. Are vSphere / Proxmox / Hyper-V in the same category? [00:28:20] Jeremy: And for people who have run their own servers with something like say VMware or Hyper V or Proxmox, are those in the same category? [00:28:32] Bryan: Yeah, I mean a little bit. I mean, it kind of like vSphere Yes. Via VMware. No. So it's like you, uh, VMware ESX is, is kind of a key building block upon which you can build something that is a more meaningful distributed system. When it's just like a machine that you're provisioning VMs on, it's like, okay, well that's actually, you as the human might be the control plane. [00:28:52] Bryan: Like, that's, that, that's, that's a much easier problem. Um, but when you've got, you know, tens, hundreds, thousands of machines, you need to do it robustly. You need something to coordinate that activity and you know, you need to pick which sled you land on. You need to be able to move these things. You need to be able to update that whole system. [00:29:06] Bryan: That's when you're getting into a control plane. So, you know, some of these things have kind of edged into a control plane, certainly VMware. Um, now Broadcom, um, has delivered something that's kind of cloudish. Um, I think that for folks that are truly born on the cloud, it, it still feels somewhat, uh, like you're going backwards in time when you, when you look at these kind of on-prem offerings. [00:29:29] Bryan: Um, but, but it, it, it's got these aspects to it for sure. Um, and I think that we're, um, some of these other things when you're just looking at KVM or just looks looking at Proxmox you kind of need to, to connect it to other broader things to turn it into something that really looks like manageable infrastructure. [00:29:47] Bryan: And then many of those projects are really, they're either proprietary projects, uh, proprietary products like vSphere, um, or you are really dealing with open source projects that are. Not necessarily aimed at the same level of scale. Um, you know, you look at a, again, Proxmox or, uh, um, you'll get an OpenStack. [00:30:05] Bryan: Um, and you know, OpenStack is just a lot of things, right? I mean, OpenStack has got so many, the OpenStack was kind of a, a free for all, for every infrastructure vendor. Um, and I, you know, there was a time people were like, don't you, aren't you worried about all these companies together that, you know, are coming together for OpenStack? [00:30:24] Bryan: I'm like, haven't you ever worked for like a company? Like, companies don't get along. By the way, it's like having multiple companies work together on a thing that's bad news, not good news. And I think, you know, one of the things that OpenStack has definitely struggled with, kind of with what, actually the, the, there's so many different kind of vendor elements in there that it's, it's very much not a product, it's a project that you're trying to run. [00:30:47] Bryan: But that's, but that very much is in, I mean, that's, that's similar certainly in spirit. [00:30:53] Jeremy: And so I think this is kind of like you're alluding to earlier, the piece that allows you to allocate, compute, storage, manage networking, gives you that experience of I can go to a web console or I can use an API and I can spin up machines, get them all connected. At the end of the day, the control plane. Is allowing you to do that in hopefully a user-friendly way. [00:31:21] Bryan: That's right. Yep. And in the, I mean, in order to do that in a modern way, it's not just like a user-friendly way. You really need to have a CLI and a web UI and an API. Those all need to be drawn from the same kind of single ground truth. Like you don't wanna have any of those be an afterthought for the other. [00:31:39] Bryan: You wanna have the same way of generating all of those different endpoints and, and entries into the system. Building a control plane now has better tools (Rust, CockroachDB) [00:31:46] Jeremy: And if you take your time at Joyent as an example. What kind of tools existed for that versus how much did you have to build in-house for as far as the hypervisor and managing the compute and all that? [00:32:02] Bryan: Yeah, so we built more or less everything in house. I mean, what you have is, um, and I think, you know, over time we've gotten slightly better tools. Um, I think, and, and maybe it's a little bit easier to talk about the, kind of the tools we started at Oxide because we kind of started with a, with a clean sheet of paper at oxide. [00:32:16] Bryan: We wanted to, knew we wanted to go build a control plane, but we were able to kind of go revisit some of the components. So actually, and maybe I'll, I'll talk about some of those changes. So when we, at, For example, at Joyent, when we were building a cloud at Joyent, there wasn't really a good distributed database. [00:32:34] Bryan: Um, so we were using Postgres as our database for metadata and there were a lot of challenges. And Postgres is not a distributed database. It's running. With a primary secondary architecture, and there's a bunch of issues there, many of which we discovered the hard way. Um, when we were coming to oxide, you have much better options to pick from in terms of distributed databases. [00:32:57] Bryan: You know, we, there was a period that now seems maybe potentially brief in hindsight, but of a really high quality open source distributed databases. So there were really some good ones to, to pick from. Um, we, we built on CockroachDB on CRDB. Um, so that was a really important component. That we had at oxide that we didn't have at Joyent. [00:33:19] Bryan: Um, so we were, I wouldn't say we were rolling our own distributed database, we were just using Postgres and uh, and, and dealing with an enormous amount of pain there in terms of the surround. Um, on top of that, and, and, you know, a, a control plane is much more than a database, obviously. Uh, and you've gotta deal with, uh, there's a whole bunch of software that you need to go, right. [00:33:40] Bryan: Um, to be able to, to transform these kind of API requests into something that is reliable infrastructure, right? And there, there's a lot to that. Uh, especially when networking gets in the mix, when storage gets in the mix, uh, there are a whole bunch of like complicated steps that need to be done, um, at Joyent. [00:33:59] Bryan: Um, we, in part because of the history of the company and like, look. This, this just is not gonna sound good, but it just is what it is and I'm just gonna own it. We did it all in Node, um, at Joyent, which I, I, I know it sounds really right now, just sounds like, well, you, you built it with Tinker Toys. You Okay. [00:34:18] Bryan: Uh, did, did you think it was, you built the skyscraper with Tinker Toys? Uh, it's like, well, okay. We actually, we had greater aspirations for the Tinker Toys once upon a time, and it was better than, you know, than Twisted Python and Event Machine from Ruby, and we weren't gonna do it in Java. All right. [00:34:32] Bryan: So, but let's just say that that experiment, uh, that experiment did ultimately end in a predictable fashion. Um, and, uh, we, we decided that maybe Node was not gonna be the best decision long term. Um, Joyent was the company behind node js. Uh, back in the day, Ryan Dahl worked for Joyent. Uh, and then, uh, then we, we, we. [00:34:53] Bryan: Uh, landed that in a foundation in about, uh, what, 2015, something like that. Um, and began to consider our world beyond, uh, beyond Node. Rust at Oxide [00:35:04] Bryan: A big tool that we had in the arsenal when we started Oxide is Rust. Um, and so indeed the name of the company is, is a tip of the hat to the language that we were pretty sure we were gonna be building a lot of stuff in. [00:35:16] Bryan: Namely Rust. And, uh, rust is, uh, has been huge for us, a very important revolution in programming languages. you know, there, there, there have been different people kind of coming in at different times and I kinda came to Rust in what I, I think is like this big kind of second expansion of rust in 2018 when a lot of technologists were think, uh, sick of Node and also sick of Go. [00:35:43] Bryan: And, uh, also sick of C++. And wondering is there gonna be something that gives me the, the, the performance, of that I get outta C. The, the robustness that I can get out of a C program but is is often difficult to achieve. but can I get that with kind of some, some of the velocity of development, although I hate that term, some of the speed of development that you get out of a more interpreted language. [00:36:08] Bryan: Um, and then by the way, can I actually have types, I think types would be a good idea? Uh, and rust obviously hits the sweet spot of all of that. Um, it has been absolutely huge for us. I mean, we knew when we started the company again, oxide, uh, we were gonna be using rust in, in quite a, quite a. Few places, but we weren't doing it by fiat. [00:36:27] Bryan: Um, we wanted to actually make sure we're making the right decision, um, at, at every different, at every layer. Uh, I think what has been surprising is the sheer number of layers at which we use rust in terms of, we've done our own embedded firmware in rust. We've done, um, in, in the host operating system, which is still largely in C, but very big components are in rust. [00:36:47] Bryan: The hypervisor Propolis is all in rust. Uh, and then of course the control plane, that distributed system on that is all in rust. So that was a very important thing that we very much did not need to build ourselves. We were able to really leverage, uh, a terrific community. Um. We were able to use, uh, and we've done this at Joyent as well, but at Oxide, we've used Illumos as a hostos component, which, uh, our variant is called Helios. [00:37:11] Bryan: Um, we've used, uh, bhyve um, as a, as as that kind of internal hypervisor component. we've made use of a bunch of different open source components to build this thing, um, which has been really, really important for us. Uh, and open source components that didn't exist even like five years prior. [00:37:28] Bryan: That's part of why we felt that 2019 was the right time to start the company. And so we started Oxide. The problems building a control plane in Node [00:37:34] Jeremy: You had mentioned that at Joyent, you had tried to build this in, in Node. What were the, what were the, the issues or the, the challenges that you had doing that? [00:37:46] Bryan: Oh boy. Yeah. again, we, I kind of had higher hopes in 2010, I would say. When we, we set on this, um, the, the, the problem that we had just writ large, um. JavaScript is really designed to allow as many people on earth to write a program as possible, which is good. I mean, I, I, that's a, that's a laudable goal. [00:38:09] Bryan: That is the goal ultimately of such as it is of JavaScript. It's actually hard to know what the goal of JavaScript is, unfortunately, because Brendan Ike never actually wrote a book. so that there is not a canonical, you've got kind of Doug Crockford and other people who've written things on JavaScript, but it's hard to know kind of what the original intent of JavaScript is. [00:38:27] Bryan: The name doesn't even express original intent, right? It was called Live Script, and it was kind of renamed to JavaScript during the Java Frenzy of the late nineties. A name that makes no sense. There is no Java in JavaScript. that is kind of, I think, revealing to kind of the, uh, the unprincipled mess that is JavaScript. [00:38:47] Bryan: It, it, it's very pragmatic at some level, um, and allows anyone to, it makes it very easy to write software. The problem is it's much more difficult to write really rigorous software. So, uh, and this is what I should differentiate JavaScript from TypeScript. This is really what TypeScript is trying to solve. [00:39:07] Bryan: TypeScript is like. How can, I think TypeScript is a, is a great step forward because TypeScript is like, how can we bring some rigor to this? Like, yes, it's great that it's easy to write JavaScript, but that's not, we, we don't wanna do that for Absolutely. I mean that, that's not the only problem we solve. [00:39:23] Bryan: We actually wanna be able to write rigorous software and it's actually okay if it's a little harder to write rigorous software that's actually okay if it gets leads to, to more rigorous artifacts. Um, but in JavaScript, I mean, just a concrete example. You know, there's nothing to prevent you from referencing a property that doesn't actually exist in JavaScript. [00:39:43] Bryan: So if you fat finger a property name, you are relying on something to tell you. By the way, I think you've misspelled this because there is no type definition for this thing. And I don't know that you've got one that's spelled correctly, one that's spelled incorrectly, that's often undefined. And then the, when you actually go, you say you've got this typo that is lurking in your what you want to be rigorous software. [00:40:07] Bryan: And if you don't execute that code, like you won't know that's there. And then you do execute that code. And now you've got a, you've got an undefined object. And now that's either gonna be an exception or it can, again, depends on how that's handled. It can be really difficult to determine the origin of that, of, of that error, of that programming. [00:40:26] Bryan: And that is a programmer error. And one of the big challenges that we had with Node is that programmer errors and operational errors, like, you know, I'm out of disk space as an operational error. Those get conflated and it becomes really hard. And in fact, I think the, the language wanted to make it easier to just kind of, uh, drive on in the event of all errors. [00:40:53] Bryan: And it's like, actually not what you wanna do if you're trying to build a reliable, robust system. So we had. No end of issues. [00:41:01] Bryan: We've got a lot of experience developing rigorous systems, um, again coming out of operating systems development and so on. And we want, we brought some of that rigor, if strangely, to JavaScript. So one of the things that we did is we brought a lot of postmortem, diagnos ability and observability to node. [00:41:18] Bryan: And so if, if one of our node processes. Died in production, we would actually get a core dump from that process, a core dump that we could actually meaningfully process. So we did a bunch of kind of wild stuff. I mean, actually wild stuff where we could actually make sense of the JavaScript objects in a binary core dump. JavaScript values ease of getting started over robustness [00:41:41] Bryan: Um, and things that we thought were really important, and this is the, the rest of the world just looks at this being like, what the hell is this? I mean, it's so out of step with it. The problem is that we were trying to bridge two disconnected cultures of one developing really. Rigorous software and really designing it for production, diagnosability and the other, really designing it to software to run in the browser and for anyone to be able to like, you know, kind of liven up a webpage, right? [00:42:10] Bryan: Is kinda the origin of, of live script and then JavaScript. And we were kind of the only ones sitting at the intersection of that. And you begin when you are the only ones sitting at that kind of intersection. You just are, you're, you're kind of fighting a community all the time. And we just realized that we are, there were so many things that the community wanted to do that we felt are like, no, no, this is gonna make software less diagnosable. It's gonna make it less robust. The NodeJS split and why people left [00:42:36] Bryan: And then you realize like, I'm, we're the only voice in the room because we have got, we have got desires for this language that it doesn't have for itself. And this is when you realize you're in a bad relationship with software. It's time to actually move on. And in fact, actually several years after, we'd already kind of broken up with node. [00:42:55] Bryan: Um, and it was like, it was a bit of an acrimonious breakup. there was a, uh, famous slash infamous fork of node called IoJS Um, and this was viewed because people, the community, thought that Joyent was being what was not being an appropriate steward of node js and was, uh, not allowing more things to come into to, to node. [00:43:19] Bryan: And of course, the reason that we of course, felt that we were being a careful steward and we were actively resisting those things that would cut against its fitness for a production system. But it's some way the community saw it and they, and forked, um, and, and I think the, we knew before the fork that's like, this is not working and we need to get this thing out of our hands. Platform is a reflection of values node summit talk [00:43:43] Bryan: And we're are the wrong hands for this? This needs to be in a foundation. Uh, and so we kind of gone through that breakup, uh, and maybe it was two years after that. That, uh, friend of mine who was um, was running the, uh, the node summit was actually, it's unfortunately now passed away. Charles er, um, but Charles' venture capitalist great guy, and Charles was running Node Summit and came to me in 2017. [00:44:07] Bryan: He is like, I really want you to keynote Node Summit. And I'm like, Charles, I'm not gonna do that. I've got nothing nice to say. Like, this is the, the, you don't want, I'm the last person you wanna keynote. He's like, oh, if you have nothing nice to say, you should definitely keynote. You're like, oh God, okay, here we go. [00:44:22] Bryan: He's like, no, I really want you to talk about, like, you should talk about the Joyent breakup with NodeJS. I'm like, oh man. [00:44:29] Bryan: And that led to a talk that I'm really happy that I gave, 'cause it was a very important talk for me personally. Uh, called Platform is a reflection of values and really looking at the values that we had for Node and the values that Node had for itself. And they didn't line up. [00:44:49] Bryan: And the problem is that the values that Node had for itself and the values that we had for Node are all kind of positives, right? Like there's nobody in the node community who's like, I don't want rigor, I hate rigor. It's just that if they had the choose between rigor and making the language approachable. [00:45:09] Bryan: They would choose approachability every single time. They would never choose rigor. And, you know, that was a, that was a big eye-opener. I do, I would say, if you watch this talk. [00:45:20] Bryan: because I knew that there's, like, the audience was gonna be filled with, with people who, had been a part of the fork in 2014, I think was the, the, the, the fork, the IOJS fork. And I knew that there, there were, there were some, you know, some people that were, um, had been there for the fork and. [00:45:41] Bryan: I said a little bit of a trap for the audience. But the, and the trap, I said, you know what, I, I kind of talked about the values that we had and the aspirations we had for Node, the aspirations that Node had for itself and how they were different. [00:45:53] Bryan: And, you know, and I'm like, look in, in, in hindsight, like a fracture was inevitable. And in 2014 there was finally a fracture. And do people know what happened in 2014? And if you, if you, you could listen to that talk, everyone almost says in unison, like IOJS. I'm like, oh right. IOJS. Right. That's actually not what I was thinking of. [00:46:19] Bryan: And I go to the next slide and is a tweet from a guy named TJ Holloway, Chuck, who was the most prolific contributor to Node. And it was his tweet also in 2014 before the fork, before the IOJS fork explaining that he was leaving Node and that he was going to go. And you, if you turn the volume all the way up, you can hear the audience gasp. [00:46:41] Bryan: And it's just delicious because the community had never really come, had never really confronted why TJ left. Um, there. And I went through a couple folks, Felix, bunch of other folks, early Node folks. That were there in 2010, were leaving in 2014, and they were going to go primarily, and they were going to go because they were sick of the same things that we were sick of. [00:47:09] Bryan: They, they, they had hit the same things that we had hit and they were frustrated. I I really do believe this, that platforms do reflect their own values. And when you are making a software decision, you are selecting value. [00:47:26] Bryan: You should select values that align with the values that you have for that software. That is, those are, that's way more important than other things that people look at. I think people look at, for example, quote unquote community size way too frequently, community size is like. Eh, maybe it can be fine. [00:47:44] Bryan: I've been in very large communities, node. I've been in super small open source communities like AUMs and RAs, a bunch of others. there are strengths and weaknesses to both approaches just as like there's a strength to being in a big city versus a small town. Me personally, I'll take the small community more or less every time because the small community is almost always self-selecting based on values and just for the same reason that I like working at small companies or small teams. [00:48:11] Bryan: There's a lot of value to be had in a small community. It's not to say that large communities are valueless, but again, long answer to your question of kind of where did things go south with Joyent and node. They went south because the, the values that we had and the values the community had didn't line up and that was a very educational experience, as you might imagine. [00:48:33] Jeremy: Yeah. And, and given that you mentioned how, because of those values, some people moved from Node to go, and in the end for much of what oxide is building. You ended up using rust. What, what would you say are the, the values of go and and rust, and how did you end up choosing Rust given that. Go's decisions regarding generics, versioning, compilation speed priority [00:48:56] Bryan: Yeah, I mean, well, so the value for, yeah. And so go, I mean, I understand why people move from Node to Go, go to me was kind of a lateral move. Um, there were a bunch of things that I, uh, go was still garbage collected, um, which I didn't like. Um, go also is very strange in terms of there are these kind of like. [00:49:17] Bryan: These autocratic kind of decisions that are very bizarre. Um, there, I mean, generics is kind of a famous one, right? Where go kind of as a point of principle didn't have generics, even though go itself actually the innards of go did have generics. It's just that you a go user weren't allowed to have them. [00:49:35] Bryan: And you know, it's kind of, there was, there was an old cartoon years and years ago about like when a, when a technologist is telling you that something is technically impossible, that actually means I don't feel like it. Uh, and there was a certain degree of like, generics are technically impossible and go, it's like, Hey, actually there are. [00:49:51] Bryan: And so there was, and I just think that the arguments against generics were kind of disingenuous. Um, and indeed, like they ended up adopting generics and then there's like some super weird stuff around like, they're very anti-assertion, which is like, what, how are you? Why are you, how is someone against assertions, it doesn't even make any sense, but it's like, oh, nope. [00:50:10] Bryan: Okay. There's a whole scree on it. Nope, we're against assertions and the, you know, against versioning. There was another thing like, you know, the Rob Pike has kind of famously been like, you should always just run on the way to commit. And you're like, does that, is that, does that make sense? I mean this, we actually built it. [00:50:26] Bryan: And so there are a bunch of things like that. You're just like, okay, this is just exhausting and. I mean, there's some things about Go that are great and, uh, plenty of other things that I just, I'm not a fan of. Um, I think that the, in the end, like Go cares a lot about like compile time. It's super important for Go Right? [00:50:44] Bryan: Is very quick, compile time. I'm like, okay. But that's like compile time is not like, it's not unimportant, it's doesn't have zero importance. But I've got other things that are like lots more important than that. Um, what I really care about is I want a high performing artifact. I wanted garbage collection outta my life. Don't think garbage collection has good trade offs [00:51:00] Bryan: I, I gotta tell you, I, I like garbage collection to me is an embodiment of this like, larger problem of where do you put cognitive load in the software development process. And what garbage collection is saying to me it is right for plenty of other people and the software that they wanna develop. [00:51:21] Bryan: But for me and the software that I wanna develop, infrastructure software, I don't want garbage collection because I can solve the memory allocation problem. I know when I'm like, done with something or not. I mean, it's like I, whether that's in, in C with, I mean it's actually like, it's really not that hard to not leak memory in, in a C base system. [00:51:44] Bryan: And you can. give yourself a lot of tooling that allows you to diagnose where memory leaks are coming from. So it's like that is a solvable problem. There are other challenges with that, but like, when you are developing a really sophisticated system that has garbage collection is using garbage collection. [00:51:59] Bryan: You spend as much time trying to dork with the garbage collector to convince it to collect the thing that you know is garbage. You are like, I've got this thing. I know it's garbage. Now I need to use these like tips and tricks to get the garbage collector. I mean, it's like, it feels like every Java performance issue goes to like minus xx call and use the other garbage collector, whatever one you're using, use a different one and using a different, a different approach. [00:52:23] Bryan: It's like, so you're, you're in this, to me, it's like you're in the worst of all worlds where. the reason that garbage collection is helpful is because the programmer doesn't have to think at all about this problem. But now you're actually dealing with these long pauses in production. [00:52:38] Bryan: You're dealing with all these other issues where actually you need to think a lot about it. And it's kind of, it, it it's witchcraft. It, it, it's this black box that you can't see into. So it's like, what problem have we solved exactly? And I mean, so the fact that go had garbage collection, it's like, eh, no, I, I do not want, like, and then you get all the other like weird fatwahs and you know, everything else. [00:52:57] Bryan: I'm like, no, thank you. Go is a no thank you for me, I, I get it why people like it or use it, but it's, it's just, that was not gonna be it. Choosing Rust [00:53:04] Bryan: I'm like, I want C. but I, there are things I didn't like about C too. I was looking for something that was gonna give me the deterministic kind of artifact that I got outta C. But I wanted library support and C is tough because there's, it's all convention. you know, there's just a bunch of other things that are just thorny. And I remember thinking vividly in 2018, I'm like, well, it's rust or bust. Ownership model, algebraic types, error handling [00:53:28] Bryan: I'm gonna go into rust. And, uh, I hope I like it because if it's not this, it's gonna like, I'm gonna go back to C I'm like literally trying to figure out what the language is for the back half of my career. Um, and when I, you know, did what a lot of people were doing at that time and people have been doing since of, you know, really getting into rust and really learning it, appreciating the difference in the, the model for sure, the ownership model people talk about. [00:53:54] Bryan: That's also obviously very important. It was the error handling that blew me away. And the idea of like algebraic types, I never really had algebraic types. Um, and the ability to, to have. And for error handling is one of these really, uh, you, you really appreciate these things where it's like, how do you deal with a, with a function that can either succeed and return something or it can fail, and the way c deals with that is bad with these kind of sentinels for errors. [00:54:27] Bryan: And, you know, does negative one mean success? Does negative one mean failure? Does zero mean failure? Some C functions, zero means failure. Traditionally in Unix, zero means success. And like, what if you wanna return a file descriptor, you know, it's like, oh. And then it's like, okay, then it'll be like zero through positive N will be a valid result. [00:54:44] Bryan: Negative numbers will be, and like, was it negative one and I said airo, or is it a negative number that did not, I mean, it's like, and that's all convention, right? People do all, all those different things and it's all convention and it's easy to get wrong, easy to have bugs, can't be statically checked and so on. Um, and then what Go says is like, well, you're gonna have like two return values and then you're gonna have to like, just like constantly check all of these all the time. Um, which is also kind of gross. Um, JavaScript is like, Hey, let's toss an exception. If, if we don't like something, if we see an error, we'll, we'll throw an exception. [00:55:15] Bryan: There are a bunch of reasons I don't like that. Um, and you look, you'll get what Rust does, where it's like, no, no, no. We're gonna have these algebra types, which is to say this thing can be a this thing or that thing, but it, but it has to be one of these. And by the way, you don't get to process this thing until you conditionally match on one of these things. [00:55:35] Bryan: You're gonna have to have a, a pattern match on this thing to determine if it's a this or a that, and if it in, in the result type that you, the result is a generic where it's like, it's gonna be either the thing that you wanna return. It's gonna be an okay that contains the thing you wanna return, or it's gonna be an error that contains your error and it forces your code to deal with that. [00:55:57] Bryan: And what that does is it shifts the cognitive load from the person that is operating this thing in production to the, the actual developer that is in development. And I think that that, that to me is like, I, I love that shift. Um, and that shift to me is really important. Um, and that's what I was missing, that that's what Rust gives you. [00:56:23] Bryan: Rust forces you to think about your code as you write it, but as a result, you have an artifact that is much more supportable, much more sustainable, and much faster. Prefer to frontload cognitive load during development instead of at runtime [00:56:34] Jeremy: Yeah, it sounds like you would rather take the time during the development to think about these issues because whether it's garbage collection or it's error handling at runtime when you're trying to solve a problem, then it's much more difficult than having dealt with it to start with. [00:56:57] Bryan: Yeah, absolutely. I, and I just think that like, why also, like if it's software, if it's, again, if it's infrastructure software, I mean the kinda the question that you, you should have when you're writing software is how long is this software gonna live? How many people are gonna use this software? Uh, and if you are writing an operating system, the answer for this thing that you're gonna write, it's gonna live for a long time. [00:57:18] Bryan: Like, if we just look at plenty of aspects of the system that have been around for a, for decades, it's gonna live for a long time and many, many, many people are gonna use it. Why would we not expect people writing that software to have more cognitive load when they're writing it to give us something that's gonna be a better artifact? [00:57:38] Bryan: Now conversely, you're like, Hey, I kind of don't care about this. And like, I don't know, I'm just like, I wanna see if this whole thing works. I've got, I like, I'm just stringing this together. I don't like, no, the software like will be lucky if it survives until tonight, but then like, who cares? Yeah. Yeah. [00:57:52] Bryan: Gar garbage clock. You know, if you're prototyping something, whatever. And this is why you really do get like, you know, different choices, different technology choices, depending on the way that you wanna solve the problem at hand. And for the software that I wanna write, I do like that cognitive load that is upfront. With LLMs maybe you can get the benefit of the robust artifact with less cognitive load [00:58:10] Bryan: Um, and although I think, I think the thing that is really wild that is the twist that I don't think anyone really saw coming is that in a, in an LLM age. That like the cognitive load upfront almost needs an asterisk on it because so much of that can be assisted by an LLM. And now, I mean, I would like to believe, and maybe this is me being optimistic, that the the, in the LLM age, we will see, I mean, rust is a great fit for the LLMH because the LLM itself can get a lot of feedback about whether the software that's written is correct or not. [00:58:44] Bryan: Much more so than you can for other environments. [00:58:48] Jeremy: Yeah, that is a interesting point in that I think when people first started trying out the LLMs to code, it was really good at these maybe looser languages like Python or JavaScript, and initially wasn't so good at something like Rust. But it sounds like as that improves, if. It can write it then because of the rigor or the memory management or the error handling that the language is forcing you to do, it might actually end up being a better choice for people using LLMs. [00:59:27] Bryan: absolutely. I, it, it gives you more certainty in the artifact that you've delivered. I mean, you know a lot about a Rust program that compiles correctly. I mean, th there are certain classes of errors that you don't have, um, that you actually don't know on a C program or a GO program or a, a JavaScript program. [00:59:46] Bryan: I think that's gonna be really important. I think we are on the cusp. Maybe we've already seen it, this kind of great bifurcation in the software that we write where the rigorous software becomes much more important. We, we have this foundational software that we're gonna rely on as much more bedrock, and then we're gonna have much more software. [01:00:06] Bryan: Where that rigor is not as more much of a constraint because the constraints are the ability for it to be, uh, customized to my need and done very quickly. And it's, it's going to feel like, I think two different worlds. Um, and I think in an exciting way, I think it's gonna be, I think that the future's definitely exciting for software. Illumos [01:00:27] Jeremy: Another interesting decision about the oxide computer is you chose an operating system that I think most people aren't familiar with rather than, rather than a Linux or a free BSD, you chose I believe it's Illumos. [01:00:44] Jeremy: Is that how you pronounce it? [01:00:45] Bryan: Yeah, yeah, yeah, yeah. I mean, yes. So I Illumos and, uh, Illumos, which very much inherits from, uh, open Solaris, um, which inherits from the, uh, Solaris heritage, which was SunOS 4.X before that was the Solaris itself is, is kind of, uh, this unholy love child of SunOS 4.X and the, the true kind of BSD lineage and the AT&T lineage in Unix and SVR4. [01:01:15] Bryan: in many ways I know that people haven't necessarily heard of it, um, but it. Is, it is true Unix in a, in, in a real historical sense, um, in a way that, that Linux and even the bsds are not. Uh, and it, and there are aspects where it shows it, but in terms of, it is an idiosyncratic decision. [01:01:35] Bryan: It's not one that we took lightly. people assume like, oh, you know, it's a bunch of, it's a bunch of old Sun folks. [01:01:41] Bryan: Like of course they're gonna pick Illumos like that's, and we had a lot of operational experience with, with SmartOS, uh, at Joyent But that's a, it's more nuance than that, I would say. it is true that a bunch of the technologies that we have developed over the years, uh, we developed them for good reasons. Debugability of the OS [01:01:55] Bryan: Um, and we don't want to be without them. I mentioned the postmortem diagnosability of JavaScript. Uh, postmortem diagnosability is really important to us, and to me, debug ability is really important. Um, and debug ability is not something that others operating systems have taken that seriously. [01:02:10] Bryan: Just to put it bluntly, you can, you, you can judge a lot about an operating system by it's built in debugger. and, uh, there's not really a built in debugger present for Linux. but it, it, it, it's more than just that. Linux is a Kernel, not an OS, so you need many things on top of it [01:02:22] Bryan: You know, uh, if you look at Linux in particular, Linux is a kernel. [01:02:27] Bryan: It is not. And you know, this is something that. Linus Torvalds makes clear that you know, in his defense, at more or less every juncture, Linux is a kernel. And thi this is the kind of the very famous, you know, the Stallman is of like what you were calling Linux. I call it GNU/Linux Linux. And it's obviously like, on the one hand, a little bit ridiculous. [01:02:45] Bryan: On the other hand, like not wrong in that the software that you need to actually have a functional system is a lot more than just Linux. You need a libc, right? And there's actually more than one libc you. Is it gonna be musl Is it gonna be glibc? There are some other alternate libcs, um, you need. And part of the problem is if you're going to use a Linux kernel as the basis for a host operating system. That's nowhere near enough because you need to now build an entire distro effectively of all these different tools on top of that. And the maintenance burden of that is off the charts. Um, and it's actually funny, one of our colleagues, Laura Abbott, came from Red Hat [01:03:26] Bryan: And I'm like, well, this will be interesting. 'cause I, again, my mind was open, more open than it had been in, you know, a long time about like, maybe we should use Linux I don't know. And Laura felt strongly that we shouldn't, but for reasons that were different. [01:03:40] Bryan: Laura was very concerned about what the, the distro management burden would be, and she had really seen that upfront at Red Hat, and she's like, do not make this decision lightly, that you've gotta make all these other decisions. It's something I really like. I remember at the time being like, okay, yeah, I know I get that. [01:03:58] Bryan: But then in a year since I'm like, oh, wow, that's a really big deal. That's a bigger deal than I realized. and so we, you know, we, we don't have to do that. We, we, we get an operating system that's got a lot of stuff like built in. Um, and now of course it means that like we've also, it, it's talk about small communities, definitely a small community. [01:04:16] Bryan: Um, and there, there you, there are, there are challenges there too. Um, but we, it, it is one that has allowed us to get the technologies that we really need. ZFS, containers, DTrace, the, the virtual networking, a bunch of things that we really needed, we had in place and then allowed us to also move the system in a dimension that we needed and wanted to move it. [01:04:37] Bryan: So, um, for us it's been the right decision. but I think, again, the internet is kind of disbelieving that we actually were at a real juncture there. But, uh, but we were, um, we, we hand on heart were, uh, actually at the moment where I was like most potentially intrigued or at least wanted to explore something like Linux. [01:04:56] Bryan: Uh, Linux Torvalds was just on an absolute bender against ZFS which he's done occasionally. And I'm like, this is, dude, this is the wrong time to be on this th this ZFS tamper tantrum, which he's again done a couple of times. Um, but it was kind of the, and, and a problem with Linux is that ZFS is not a first class, first class file system. [01:05:16] Bryan: Um, has always been on the outside because of this kind of, these ridiculous licensing concerns. And it was a reminder of like, oh yeah, right this yeah okay i think yeah we'll go our own way. Illumos and FreeBSD include a lot more making it easier to manage a distribution [01:05:28] Jeremy: And you, you mentioned one of the, the risks would be managing your own distribution, but for Illumos you actually did create your own distribution. [01:05:38] Bryan: Yeah, you do, but like it's yes and no. I mean, you are creating your own distribution, but you're not having to like go be like, okay, God, we gotta go get our own version of GDB. We gotta go get our own version of grep, we gotta get our own version of tar, we gotta get our own version. And it's like you, that's what I mean about like in Illumos and FreeBSD is like this too, by the way. [01:05:55] Bryan: It's not the only operating system like this. Actually Windows is like this. It is the operating system is more, is the kernel plus the system libraries, plus the commands to run So you are, you're not having these things operate across purposes. You've got, and technologies like DTrace that are able to straddle this user kernel boundary in a really important way. [01:06:16] Bryan: So that's what I mean by a distro. we, we've got our, our own derivative. Um, but we actually very deliberately in part because of our experiences at Joyent, uh, we really get stuff upstream. So we are as, as much as we humanly can, everything is, is upstream. The other thing we knew we wanted to go do, or had a hunch we wanted to go do, we knew that we wanted to eliminate the BIOS completely and for all of the things that we did at Oxide, which is we did our own switch. [01:06:45] Bryan: We did our own compute sled. We did, you know, that, that our own microcontroller operating system in many ways, the riskiest thing was this idea of eliminating the BIOS. And so we knew one of the things we wanted to go do was actually do that lowest level of platform, initialization ourselves from the actual operating system, from the host operating system. [01:07:05] Bryan: And that required us to get this thing really under a knife. we've managed to pull that off. But that would've been much thornier, I think, trying to kind of battle, um, in upstream, because it's a very different architecture. what we have is not a PC architecture, it's an x86 machine, AMD. [01:07:23] Bryan: But we have done, we in the host operating system have done the lowest level of platform neutralization that's historically done by the BIOS. Um, which was a huge challenge. And one by the way that AMD didn't think we could do, AMD was very surprised when we were, you know, sometimes you know that somebody is like not explicitly underestimating you, but then they're very surprised when you've succeeded and you realize like, oh, okay, yeah, you actually, I guess you were underestimating me. [01:07:48] Bryan: I didn't realize that. And that was definitely the, the case here where, uh, AMD was very surprised that we lived, um, and that we actually had a machine that could boot, um, without running a UEFI BIOS. But that, that very much, Was hand in glove with this decision to have our own Illumos derivativefor Helios. [01:08:05] Jeremy: And it, it sounded like possibly for this to work, you would have to upstream things to Illumos. Is that right? [01:08:13] Bryan: I mean, for sure. You just don't wanna be you, you don't wanna be battling a, you are gonna end up with your own kind of derivative at some level. And this is where it's actually just helpful to have a smaller community, frankly. Um, just makes it easier to go do that stuff. Upstreaming to Linux is a fight and rust is a battlefield [01:08:28] Jeremy: Yeah, because with Linux having so many users, it would be much more of a fight trying to, yeah. [01:08:36] Bryan: Oh everything's a fight. I mean, everything's a fight is the problem. I mean, uh, and you get like, I mean, and I'll tell you another thing, it's actually, this is like really funny because, uh, this is is something that, um, that that Torvalds and I actually agree on, uh, pretty thoroughly, which is what the, the presence of rust in the system. [01:08:52] Bryan: so we've been able to add, I mean, I'm absolutely believed that like we want to have rust internal modules and we do in, in, in Helios. We've, we've got, rust in kernel modules in terms of, of our oxide packet, transformation engine, OPTE. and we, that will increase over time. That is an absolute civil war in Linux ironically, one that like, again, Torvalds and I agree, but there are people, other people in Linux who ardently disagree over ardently, disagree over rust in the kernel and like I, that is not a fight I want to have. [01:09:26] Bryan: Like I'm just, I am. Absolutely disinterested in having an argument over Rust versus C. Like, no, thank you. that is the peril of a larger community is you end up with people and then, you know, a larger community that like look is not exactly an exemplar of, of positive community behavior. Um, and there are plenty of people that have been, uh, that have done Linux kernel development and then have walked away from it because it was not exactly an uplifting experience. [01:09:52] Bryan: And, uh, their treatment of rust is really bad right now. I mean, I think that's like one of the big open challenges that I think Linux has, what does the future of the system look like with respect to Rust? I feel it's like, it's the Torvalds position is the only one that's ultimately tenable in the limit, but clearly there are people inside of Linux maintainers inside of Linux that hardly disagree with that. [01:10:11] Jeremy: And for those who aren't familiar with the fight, what is, what is his [01:10:16] Jeremy: opinion versus the, [01:10:17] Bryan: I mean, Torvalds I think is, is Torvalds is Pro Rust. Is, is is, I, I think is the, the, the short answer. I shouldn't speak for him obviously, but, but my read on this from the outside looking in is Torvalds is broadly like, yeah, this is like, this is interesting. The same reasons I think it's interesting and that we should have kernel components written in rust. [01:10:33] Bryan: Um, but you've got, um, I think maintainers of sub of subsystems that have, are ardently opposed to it and are kinda holding aspects of the hostage of the project hostage over it. there's gonna be, there's gonna continue to be friction, um, uh, over the, the, the future of the operating system there. [01:10:53] Bryan: And like I. I don't want to have debates. I really don't. I mean, I, I, I, I guess maybe, you know, okay, maybe I, I, I, I do kick an occasional hornet's nest on the internet, but I, I really like, don't wanna be spending, I, I would much rather find a community where we're gonna broadly agree on our values and then we, than I just find that there is, that there is, much less discord, um, when you, you have that kind of, that value alignment. [01:11:21] Bryan: Um, and I think that that's one of the big challenges for Linux is there is a, there is, um, some, some, uh, disalignment there, uh, in terms of what people view as the future of the operating system. [01:11:33] Jeremy: And, and this choice of your operating system as a user of an oxide computer, would they [01:11:40] Bryan: No, you don't know. I mean, it, it's just like you don't know, like, you know, if you're a user of AWS you've got no idea what Nitro is or Annaperna or these other things that are like, you know, do you know whether they use KVM at AWS? No, you've got no idea. I mean, the, do they use KVM or Zen? It's like, the answer is like, well, it depends on your instance type and the age of the thing. [01:12:00] Bryan: And, but, but like, as a user that's, that, that's opaque to you. Um, what you've got is like, you wanna hit an API endpoint, you wanna provision an instance and you wanna run the, uh, a, an operating system of your choice. You wanna run Windows or Linux or, or, you know, or previous SD or whatever as a, in your cloud. [01:12:19] Bryan: That's what you actually want to go do. Um, and that's what we enable you to do. FPGAs, ASICS, etc [01:12:25] Jeremy: Something I find interesting about the OXIDE project is you not only write software that's general compute to run on x86. There's also parts of the system that are written for FPGAs and written for ASICS I wonder if you could explain to people who might not be familiar what the difference is and what it's like building for those things versus running something on a a VM on an x86 machine. [01:12:59] Bryan: Yeah, I mean, so you, you've got, I mean, there's a whole spectrum of, and, and this is all kind of constitutes the, the platform for compute. I mean, you have the x86 cores, uh, AMD cores, which are the, I mean, that, that's the kind of compute that you know about, right? That, that is kind of architecturally present in terms of the compute that's elsewhere in the system. [01:13:16] Bryan: I mean, you've got microcontrollers, so we talked with a baseboard management controller. Uh, we have turned that into a much smaller thing, a service processor, that, that's a microcontroller, a, a Cortex m uh, microcontroller that is running that, that you don't see as a user. Um, but we've got our own system software, appropriately enough called Hubris. [01:13:36] Bryan: Um, that, that, because we actually thought we were not gonna use our, we thought we were gonna use a, a, an a, an open source operating system, but then realized that actually we needed to go our own way. and the, the debugger for hubris is appropriately enough, called the Humility. Um, so y the but hubris runs on that microcontroller, it runs on our root of trust microcontroller. [01:13:56] Bryan: Um, you, but that you also have, um, we've got FPGAs in the system. Um, FPGA is doing different kinds of things and FPGA is a field programmable gate array and you also call, you'll hear it referred to as soft logic. And, uh, FPGAs are really neat because they are, uh, they are hardware that you can reconfigure dynamically. [01:14:16] Bryan: And the thing that's really cool about that is we, we can take our FPGA payload and put it into a hubris, larger hubris payload, and then you can sign that and attach to that. And so now you, you, you've got confidence about that whole bundle that you're loading onto the system. And then when we start the system, we will actually load this Bitstream onto the FPGA. [01:14:38] Bryan: You know, for in, in our different platforms, the FPGAs have different roles, but they're doing low level things like power sequencing. They're doing, we generally are using FPGAs. Historically we're using FPGAs to do this kind of, this, these low level things on the board. 'cause it makes for a simple board, right? [01:14:54] Bryan: It makes for a simple board when you can just put the soft logic down. Um, and some of these are crazy small FPGAs, that are, are small, but they're, there are enough for what we need. we also have higher performance FPGAs that we are increasingly using, uh, for things like NICs uh, network interface cards. [01:15:11] Bryan: So we are. some of the, the kind of the next gen FPGAs, are really pretty cool because they have what you call a hard block, which is say kind of like one of the components that the, the hard components on. They're not programmable. That allows you to have like a surveys that allows you to actually speak over a network, which is actually a really hard part of a nick. [01:15:30] Bryan: And then you could do the NIC in soft logic, which is really neat. Um, so we're really excited about that. That's the kind of in a forthcoming platform. We're working on a, uh, an FPGA there. Uh, and then of course we've got, so in asic uh, um, is an application specific integrated circuit, and that's, uh, not dynamically reconfigurable. [01:15:51] Bryan: Um, and ASIC is is where, and we've got any number of asics on the system where you've got a, a deliberately fabbed part. and so we, you know, like our traditional NIC is an ASIC Um, and we haven't done any ASIC development yet at oxide, but I think any oxide engineer would say would emphasize the yet, uh, that at some point we almost, we'll do an asic. [01:16:12] Bryan: And ASIC is still really, really hairy to do. Um, FPGAs are great because they look a lot like software. Asics are a lot more complicated and for a bunch of reasons that are, that are owed, that's they're that own kinda like deep well of complexity. Um, but uh, not least the software licensing for an ASIC is brutal. [01:16:32] Bryan: 'cause you actually need to get EDA tools from the likes of synopsis or cadence and, you're kind of mating yourself onto a fab in a process. And it's like, there's a whole, like, it's very, very involved to do an asic, very expensive to do an asic. It's higher performing, consumes less power. It's cheaper when you're actually, when you actually have the artifact. [01:16:49] Bryan: Uh, and there are reasons why we, you know, we can envision ourselves doing asics in the future, but for right now, the asics we buy are all off the shelf. and then we use FPGAs in a bunch of different places of the system. it, I, I would, uh, refer folks, you know, we've got an Oxide and Friends podcast. [01:17:03] Bryan: We actually have, uh, Raja Kodori um, joined us, uh, for one of our episodes talking about exactly this, about these different kinds of compute elements. And that's another, a great episode that I would steer people to if they wanted to learn more about the different kinds of compute. The Oxide Computer [01:17:17] Jeremy: The oxide machine itself, I don't think we've really explained physically what it is. You know, what are people going to be receiving, what are the components, what is it, how is it different? [01:17:31] Jeremy: Yeah. What, what would I actually be buying and, and getting into my data center. So maybe you could talk a little bit about what that [01:17:38] Bryan: Yeah. So what, so, uh, you get a crate, um, you get a very large crate. Um, and in that crate is a rack and that rack is a, uh, it's an oxide designed rack. Um, and that rack has two switches and 32 compute sleds that are in it. Um, and that rack has been designed holistically. So we have designed all these parts to work together. [01:18:03] Bryan: We've got some very basic things. We've got a DC bus bar in the rack. So we've got a power shelf that rectifies from AC to DC and then we run DC up and down the back of, of the rack and our compute sleds blind mate into that DC power. That is basic. And anybody at all of the hyperscalers have a DC busbar based system. [01:18:27] Bryan: You can't buy a DC busbar based system from Dell, HP, Supermicro because in order to have a busbar based system, you really need to be designing at at least the rack level because the rack needs to be the, you can't just buy an oxide sled. Um, you need to buy the rack that is gonna, is gonna contain it. And then we very much designed the rack around the switch. [01:18:53] Bryan: So the switch, we actually have a cable, the back plane, and in addition to blind mating into power, our compute slots blind made into networking. So the networking is a passive cabled back plane, and it means there's no actual cabling in the sled itself. So if you wanna take a sled out, you just like take a sled out, there's no cables. [01:19:13] Bryan: One of the challenges with the kind of traditional server design and the baseboard management controller, it's gotta be on two different networks. Well, ours is on two different networks too, but you don't see any of those because it's all sitting on that cable back plane. [01:19:23] Bryan: And our switch, which has that intel tofino at its core, has a, there's actually another switch that that serves as the switch for the service processors on the system. And then that's all, that's all, uh, coherently interconnected. Um, so what you get is a real product and that thing wheels into the data center. [01:19:41] Bryan: You get power applied. And you are on the, the, the tech port doing the basic configuration we need, which basically like we need to know, like who to talk to for time, more or less. We gotta find, you know, there, there's some real basics that we need. We need a BGP session and we need to be able to actually like connect to a network. [01:19:59] Bryan: But once we can do that, we're off. And you're provisioning VMs. Uh, you're provisioning storage, you're provisioning networks, virtual networks, just like you're on the public cloud. And one of the things that's been really, really fun, because this is really painful with traditional infrastructure, it's, it's this kind of bag of bolts approach can take a long, it can take a comically long time. I'm talking like easily months to get the gear has arrived. Get to the point where devs are on there. You're like, what takes months? It's like, well glad you asked. It's like we got the wrong switch from this vendor. This, that server came with the wrong rack rails, we're missing the cabling over here. We got it all together and the software vendor, the software didn't work and the software vendors pointing at the, so it's like these things just like can take a long time to get up and running with the oxide rack. [01:20:48] Bryan: That thing wheels in, you power it on, you plug it in, you get a BGP session and your provisioning of VMs. And one of the thi the experiences that's been really fun is to be with customers as they are doing that for the first time and be like, oh my God, I'm on the, like we're provisioning infrastructure. [01:21:05] Bryan: Like, and I can just like. Let my devs at it. Like we, we just got this thing powered on like two hours ago and that is really, really neat. Um, and very vindicating of the approach that we've taken. It's really exciting. I. [01:21:21] Jeremy: And for a, a listener who's never had to do it the traditional way, what are all the different things that they would need to get into this rack to have the [01:21:31] Bryan: They would need, you gotta go buy, switch from a switch vendor. You gotta go buy, you're buying your compute from someone else. Maybe you're buying storage from someone else. You're cabling it together. You're get you, you've got, you've got your service. Your BMCs gotta be on a network. So you got a separate switch for that. [01:21:44] Bryan: You've gotta have, and then you, that's all, that's the hardware you get. Get software on it. Like, what are you doing for, are you getting OpenStack? You're doing VMware, are you doing something else? And it's like, then that's all gotta work. You've gotta get that image on there. You've gotta, and it's like you're just kind of building this thing as you go. [01:21:58] Bryan: I mean, there's just nothing that is, you're the one kind of literally cobbling this thing together. Um, and there it's just not a product experience. [01:22:08] Jeremy: Yeah, and I think the interesting part was something you talked about earlier too, is that if something isn't behaving the way you expect, for example. IO latency or maybe some kind of compute issue it, it seems like in most of those cases, you really have no one to turn to. You can go to Dell or HP and they'll probably just shrug. [01:22:32] Bryan: Um, the, well, yeah, you know, there was, there was kind of a, a, a tweet when we were starting the company that, that we were all, I I, I think may have made it into a slide deck or two, uh, where someone said that they, they were, uh, dealing with a, a support issue with Dell and EMC, and they felt like they were talking to their divorced parents. [01:22:51] Bryan: It's not necessarily malicious because like a lot of the problem is that like you have these systems that are each well designed on their own, or they're well designed, maybe that's putting, that's, that is giving them way too much credit. [01:23:04] Bryan: They're designed on their own and then they have an issue when you put them together. That each can plausibly blame on the other. Where there, you know, just like anything where you got like, you know, miscommunication with someone or what have you, where it's like, well, you know, you thought I was to blame and I thought you were to blame. [01:23:22] Bryan: And I don't know, it's kinda like, it, it, it's in the middle, right? And this happens a lot where you have, it's like actually these two things are kind of behaving somewhat reasonably in on their own, but you put it together and it, it, like the, the result is terrible. And the problem is, it is the end user that is left being like, well, who helps me in my terrible problem? [01:23:44] Bryan: It's like, great, you each have made a convincing case that it's not your problem. Like, so I guess it's who's, I guess it's my problem I guess. Um, and that is what is really really frustrating to people. [01:23:56] Jeremy: And I, I'm curious, back when you were at Sun and, and Sun was selling their own servers, was like if somebody had a problem with one of those servers, was that a case where they could go to Sun and then Sun would actually go and figure out, okay, what's the problem here? [01:24:16] Bryan: Uh, that was, uh, on its best days. Yeah. On its best days. Uh, its days. Were not always its best days. certainly the vision that, I always had for Sun Systems is that you'd be able to do that. It was not always the case. uh, I, I started a storage group inside of Sun, uh, called Fishworks, um, in 2006. [01:24:35] Bryan: And we, we. partly wanted to really make good on this idea of systems thinking. The, uh, in, in Fish the Fish in FISHworks stands for fully integrated software and hardware, and we learned a whole set of painful lessons, um, because we were still relying on quote unquote commodity hardware there and got bit by a bunch of firmware we didn't control. [01:24:58] Bryan: Um, so I, throughout my career, I have tried to make good on this promise, but I really feel it's only at oxide that we've been able to control our, our fate sufficiently from top to bottom to really make good on it. Um, and it shows in the experiences that people are having, which is really great. [01:25:16] Jeremy: And it sounds like maybe that's primarily because the software running on each of these components is in your control, so you're able to trace through and see where the actual problem is, rather than having all these proprietary vendors. [01:25:33] Bryan: It is, yes, it is. It is that for sure it is that we just like have more of the components that our own control. don't wanna understate the importance of that. [01:25:41] Bryan: Um, I think it's also just the sense of responsibility that we take, that we, like our view is like, we're gonna own this problem no matter what. And the, you know, we've had now a couple of concrete examples where we have customer issues that, like at the end is like not a. You know, is it an oxide problem? [01:26:02] Bryan: It's like, well, no, it fill in a gap between oxide and a system we were talking to. And I think it's a point of pride that we take responsibility for taking that all the way to, to root cause. I sorry to be plugging around podcast here, whenever the, the internet jokes that we're not, that we are actually a podcasting company that is merely creating computers for content creation. [01:26:23] Bryan: And I'm not sure they're wrong, Uh, and so every, you know, whenever we have any kind of like crisis at Oxide in terms of debugging a problem or what have you, we're like, God, this is gonna be great content. And we had one of these, uh, an episode that we called Hell is other networks where, uh, we, um, had a customer. [01:26:40] Bryan: Where we took this particular switch and, and uh, talking to, to the oxide rack and each, the oxide rack and the router were, were making like reasonable decisions on their own, but you added it up and it was a nightmare. um, you know, we were able to get that debugged to root cause [01:26:56] Bryan: And even though you, again, like our behavior was reasonable and its behavior was reasonable, um, but we wanna be able to take responsibility for that. So I think it's a, it is twofold. It's the fact that we can control these layers of the stack, but then it's also very much the sense that we have suffered at the other end of this. [01:27:10] Bryan: And what we wanna be able to do is really take on that responsibility and get a customer actually righted wherever the problem may lie. [01:27:21] Jeremy: I think that's a good spot to end it on, but anything else you wanted to mention? [01:27:26] Bryan: No, it was great. I know I be hit on a bunch of things, so hopefully the, thanks for the wide ranging conversation. This is, this was, this was great. Started to go on and probably went on a couple, one too many rants, but, you know, as I'm want to do [01:27:37] Jeremy: That's what the people wanna hear. [01:27:39] Bryan: it is what the people want is what you gotta give the people what they want. [01:27:42] Bryan: You gotta give the people what, you know, what I actually didn't do at all as I did, I didn't go on any anti Oracle screed, which is now like the, I, I, uh, I, you know, am somewhat internet famous for my, my unvarnished opinions on Oracle. And, uh, you know, as, uh, as the, uh, David Ellison in Paramount has been more in the news, I've had journalists talking, calling me up because they see Hacker News comments where I'm talking about Oracle. [01:28:06] Bryan: So it's like, you know, I, I'm, this is what I'm, I'm famous for now. It's my, my unvarnished opinions and Oracle. So I'm sorry to not give the people what they want in that regard, you know? I'm sorry. [01:28:14] Jeremy: You'll, you'll always be tied to, to Larry, I guess. [01:28:19] Bryan: Minute 33 I know is always gonna, it's fine. I'll, I'll take it. Truth to power. I, I don't mind speaking it. [01:28:27] Jeremy: All right, So if people wanna check out what's going on at Oxide, check out your podcast, [01:28:32] Bryan: You have Oxide and friends. Um, you can join us live in the Discord I would love to have people, uh, join us live. Um, but, but check out the podcast feed, um, and check out the rfds. You can. Those are all, uh, out there as well. uh, and then check out the repos. Um, you know, we're, we're, everything we've talked about today is open source, so, um, people can dive in there and see what we're actually doing. [01:28:52] Bryan: we are really pride ourselves on that transparency. So, we leave nothing to the imagination. I think that sometimes people are like, okay, maybe you guys could share just a little bit less, you know, so maybe transparency to a fault, but, um, there's a lot to go dive into if you wanna, dive into oxide. [01:29:10] Jeremy: Yeah, I, I think for the people who are actually managing these data centers and working with traditional vendors, transparency is exactly what they wanna hear. [01:29:21] Bryan: It is, it is. Yes. I think that that transparency is, we did have an early customer that had an issue and we wanted to get back to them and let them know like, Hey, we're working on your issue. The customer's like, no, no, I know you're working on it. I'm watching the GitHub issue actually. [01:29:35] Bryan: It's great. It's like, oh, okay. So, you know, the, it, it's, um, we love that we, we love the fact that people can see what we're working on. We have always felt there's much more to gain than there is to lose [01:29:48] Jeremy: Very cool. Well, Bryan, thank you for chatting with me today. This was fun. [01:29:51] Bryan: Absolutely. Thanks for having me. This was a, this was a lot of fun. Uh, and, and sorry to take you into the filth that is the modern data center.

Inside a recent episode

Mayra Navarro on Getting to RubyConf (RubyConf 2023)

Published 30 Nov 2023 · Transcript excerpt

[…] I feel less insecure than the last time that I was here. Actually, my first Rubicon was Rubicon Mini in Rhode Island. So this is like the Rubicon, not the real one. It's just more, it's different. Yeah, yeah. But at the same time, it's closer. That's how I feel it. I mean, this is my fifth conference. My first one was in Colombia, RubyConf Colombia. And I got a ticket as a scholarship. But until now, I can say that it's like the feeling of the Ruby community, not only Ruby on race, Ruby community is like a... It's really positive. It has changed my life so much since the first time that I joined a community that is, I'm so happy to be a developer instead of what, you know, everybody switched jobs. […]

Who has been a guest on Software Sessions

Names identified in recent episode analyses. Showing up to five guests.

• Brandon Liu
• Hong Minhee
• Kacper "Prefetcher" Staroń
• Tom MacWright
• Paul Frazee

Reach and audience

Public platform figures. Ratings count people who left a rating, not total listeners.

Apple Podcasts (US)

5.0 / 5

3 ratings

Spotify

4.8 / 5

4 ratings

Podcast Authority Score: 29 / 100

A composite of feed quality, social presence, YouTube performance and engagement. Read the methodology.

Quality

46

Social presence

0

YouTube

0

Engagement

32

Questions about Software Sessions