Pioneer Park Podcast

About Pioneer Park Podcast

Interviews, conversations, and commentary on the frontier of science and technology gilbertgravis.substack.com (https://gilbertgravis.substack.com/s/pioneer-park-podcast?utm_medium=podcast)

Publisher

Bryan

Category

technology

Language

en

Explicit

No

First episode

4 Feb 2023

Latest episode

21 Apr 2026

Latest episodes

13 episodes in the feed.

1. 

   21 Apr 2026

   The End of Clicking? Tambo Is Reinventing User Experience | ft. Michael Magán

   What if software no longer required clicking, menus, or even learning how to use it? In this new episode, we interview Michael Magán — a technical product leader and Co-Founder of Tambo, with a background building AI/ML products at Indeed, Convoy Inc, and TaxBit. Michael shares how AI is fundamentally transforming user experience (UX)—moving from rigid interfaces to systems that understand intent. Instead of navigating apps, users can simply describe what they want, and the UI adapts in real time. If you're building, designing, or thinking about the next wave of technology, this episode is for you. Check out Tambo! https://tambo.co/ (https://tambo.co/) Follow Michael here: LinkedIn: https://www.linkedin.com/in/mnmagan/ (https://www.linkedin.com/in/mnmagan/) X: https://x.com/mrmagan_ (https://x.com/mrmagan_) Follow our socials: LinkedIn: https://www.linkedin.com/showcase/108030910 (https://www.linkedin.com/showcase/108030910) Instagram: https://www.instagram.com/pioneerparkprogram/ (https://www.instagram.com/pioneerparkprogram/) TikTok: https://www.tiktok.com/@pioneerparkprogram (https://www.tiktok.com/@pioneerparkprogram) X: https://x.com/pioneer_park_ (https://x.com/pioneer_park_) 🔔 Don’t forget to like, comment, and follow for more conversations on the future of work, technology, and innovation. #AI #UserExperience #Tambo #OpenSource #TechPodcast

2. 

   24 Mar 2026

   AI Is Still a 'Slop Machine' | Ft. Casey Rodarmor

   On this episode of Pioneer Park, Casey Rodarmor — creator of filepack, just, ordinals, contributor to Bitcoin Core, and dedicated rustacean — shares the underlying principles that motivate the projects he works on, his honest take on the future of programming in the age of AI, and lots of other hot takes. Follow Casey on X: https://x.com/rodarmor Follow his work on Github: filepack: https://github.com/casey/filepack just: https://github.com/casey/just ordinals:https://github.com/ordinals/ord Follow Pioneer Park on LinkedIn: https://www.linkedin.com/showcase/108030910 🔔 Don’t forget to like, comment, and follow for more conversations on the future of work, technology, and innovation. #AI #Programming #SoftwareEngineering #OpenSource #TechPodcast

3. 

   11 Mar 2026

   Is AI Turning Developers Into Managers? | Ft. Nick Hirsch

   AI is transforming software development faster than ever. In this conversation, Bryan sits with Nick Hirsch — tech leader, founder, and AI engineer at Tribe AI to talk about the future of AI-driven software development. As tools like ChatGPT and coding agents improve, developers are moving from writing every line of code to directing and managing AI systems that build software. They discuss how vibe coding, AI programming assistants, and agentic development are changing productivity—allowing developers to generate tens of thousands of lines of code in a single day and complete projects that once took months in just days. If you're interested in AI coding, AI developers, programming with AI, and the future of software engineering, this conversation dives into what’s happening on the frontier of applied AI. Follow Nick on LinkedIn https://www.linkedin.com/in/nickhirsch/ (https://www.linkedin.com/in/nickhirsch/) Follow Pioneer Park on LinkedIn https://www.linkedin.com/showcase/108030910 (https://www.linkedin.com/showcase/108030910) 🔔 Don’t forget to like, comment, and subscribe for more conversations on the future of work, technology, and innovation. #AIcoding #SoftwareDevelopment #ArtificialIntelligence

4. 

   18 Feb 2026

   THE SaaS APOCALYPSE: WHAT AI MEANS FOR PMs, ENGINEERS, AND FOUNDERS | ft. Ben Shoemaker

   Ben Shoemaker, a product and data leader building at the edge of agentic engineering, joins Pioneer Park to unpack the “SaaS Apocalypse” and what it means for PMs, engineers, and founders. Now a technical co-founder, Ben shares how AI coding agents and large language models are collapsing implementation costs, reshaping product roles, and making software feel increasingly disposable. We explore defensibility in an AI-first world, the rise of vibe coding and agent workflows, and the broader economic impact of commoditized intelligence. If you’re building products or navigating the future of work, this episode offers a candid look at what creating in 2026 really feels like. Follow Ben on LinkedIn https://www.linkedin.com/in/benshoemaker000/ (https://www.linkedin.com/in/benshoemaker000/) Follow Pioneer Park on LinkedIn https://www.linkedin.com/showcase/108030910 (https://www.linkedin.com/showcase/108030910) 🔔 Don’t forget to like, comment, and subscribe for more conversations on the future of work, technology, and innovation. #AI #SaaS #ProductManagement #SoftwareEngineering #Startups #AgenticAI #FutureOfWork

5. 

   25 Nov 2025

   FROM DATA SCIENTIST TO AI ENGINEER: ADAM SMITH ON THE FUTURE OF BUILDING WITH AI

   Adam Smith, an AI engineer and expert in building production-grade generative AI solutions with over 15 years of experience across multiple industries, joins us on Pioneer Park to explore the transformation of software engineering in the age of generative AI. From his early days as a data scientist building machine learning models to pioneering AI-driven solutions as a freelancer, Adam shares his unique perspective on how the role of engineers has evolved and what it means to be an “AI engineer” today. Whether you’re a software professional, AI enthusiast, or just curious about the future of technology, Adam’s insights reveal how humans and AI are collaborating to solve ambitious problems and redefine what it means to engineer in 2025. Follow for more for more deep dives into AI, technology, and the future of work. #AIEngineer #GenerativeAI #SoftwareEngineering #MachineLearning #TechInnovation #FutureOfWork #LargeLanguageModels

6. 

   4 Nov 2025

   Are You Really Ready to Be a Founder? | Ft. Logan Yonavjak

   What really makes a founder ready to lead? In this episode of Pioneer Park, Logan dives deep into what “founder readiness” truly means — beyond pitch decks and funding rounds. She explains how the Founder Readiness Institute helps founders and investors assess leadership capacity, resilience, and coachability — the human side of building a company. From balancing “zaniness” with self-awareness to learning how to handle stress, feedback, and growth, this conversation explores the psychology behind startup success. If you’ve ever wondered what separates good founders from great ones, this one’s for you. 👉 Learn more about the Founder Readiness Institute: founderready.io 💡 Follow for more conversations on leadership, innovation, and founder growth. #Entrepreneurship #Leadership #StartupFounders #FounderReadiness #Innovation

7. 

   22 Sept 2025

   How SureSteps Empowers Field Teams with Unified Knowledge | Ft. Ziwen Deng

   In this episode, we dive into the story behind SureSteps—a platform designed to help field teams work smarter, not harder. What started as a step-by-step troubleshooting tool has grown into a powerful system that makes it easier to capture, share, and transfer knowledge across industries.Ziwen Deng joins us to talk about how SureSteps is transforming the way skilled workers access information in real time, making complex processes simpler and training the next generation of field professionals faster.Whether you’re in the trades, industrial work, or any hands-on profession, SureSteps shows how unified knowledge can reduce errors, boost efficiency, and empower teams on the ground.🔗 Learn more: https://suresteps.io (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbmhhdFdIZDF0eVNuVWt2V25tUFBsdFpwNlR0Z3xBQ3Jtc0ttRXdtVWtoODdhbWlwN1dJelhDLUQ0cXFGQmZQai0xYUg0U2M5dkNHU201ejhVMk81cW8wWGY1djJBTW9sS1lBQmVTazhOeGMxU3Z0eDUzd1pkNDBRR2ZURHlFdE8tNkxyakplY20yVWZ5SUU0anZRSQ&q=https%3A%2F%2Fsuresteps.io%2F&v=-9TTq8v3UK8)📌 Don’t forget to like, comment, and subscribe for more conversations on the future of work, technology, and innovation.#FieldTeams (https://www.youtube.com/hashtag/fieldteams) #UnifiedKnowledge (https://www.youtube.com/hashtag/unifiedknowledge) #SureSteps (https://www.youtube.com/hashtag/suresteps) #FutureOfWork (https://www.youtube.com/hashtag/futureofwork) #Trades (https://www.youtube.com/hashtag/trades) #IndustrialTech (https://www.youtube.com/hashtag/industrialtech) #knowledgemanagement (https://www.youtube.com/hashtag/knowledgemanagement)

8. 

   8 May 2023

   The limits of human-derived mathematics with Jesse Michael Han

   Pioneer Park interviews Jesse Han, co-founder of Multi AI. Jesse discusses his background, experience at OpenAI, and his philosophy towards research. He draws inspiration from Alexander Grothendieck's philosophy of listening to the universe and arranging theories accordingly. He also talks about the differences between research and startup thinking, the potential for machines to inspire new algorithms, theories, and results in mathematics, and the use of language models and compute to reduce the risk of misaligned outputs. He believes that language models will become as cheap and accessible as microprocessors, and that the value will go to those who build the software and infrastructure to make them accessible to end users. He recommends that those looking to shift their career in the direction of deep learning and generative AI should work hard, find good mentors, and aim for something that will endure. Transcript Jesse Han === [00:00:00] Bryan Davis: Welcome to Pioneer Park. My name is Bryan Davis and this is John. Today we're interviewing Jesse Han. Jesse Han is the co-founder of Multi AI, an AI startup based in San Francisco. He holds a PhD in mathematics from the University of Pittsburgh, and previously worked as a research scientist at OpenAI. I know Jesse through working for Multi, for a few weeks last year, in which I was helping out with some of their product launches. And was thrilled to invite Jesse to talk a little bit more depth about his background, his experience at OpenAI and Multi as it goes forward. Welcome, Jesse. Jesse Han: Thanks for having me on the podcast, guys. Thrilled to be here. Alexander Grothendieck --- John McDonnell: Yeah. So Jesse, like I wanted to start this off by asking about on your personal webpage you have a picture of you looking up like very thoughtfully at this picture of Alexander Grothendieck. So I was curious what that picture means to you. Jesse Han: What that picture means to me. So I just thought that picture of him was really funny, [00:01:00] Because it's like this shrine. So for context, that hangs in Tong Long University in Vietnam, which was founded by one of the students that he mentored when he visited Vietnam during his career. And this was during a time that Vietnam was being bombed and he wrote some, like very moving recollections about how he would teach at the university. And then they would all have to dive into an air rate shelter and they would come out and one of the mathematicians had been like, hit by the bombs. And , that student became a very prominent mathematician in Vietnam, and he had such a lasting influence that they made this, the shine and honor of him. There's this nine foot tall portrait of him there. And I just thought it would be funny to kinda like Adam and God . But the other thing is that I take a lot of inspiration from his philosophy towards research. There's a saying that he has, a saying that he was famous for, which I think is still relevant for people working in startups today or like trying to run a company, which is that so he says that the, so I'm paraphrasing, but the mark of a good researcher is someone who listens very carefully [00:02:00] to the voices of things. They try to listen to what the universe is trying to tell them about the structure of the universe. And they they arrange their understanding and their theories and what they're doing accordingly. And I think similarly when you are trying to build something, when you're trying to do something new, you have to listen to what the world is telling you. You have to listen to what the market is telling you and build accordingly. I hope that was philosophical enough for you, . John McDonnell: Yeah, I love it. There's this there's this guy David Whyte, who's who's a poet, and he has this kind of concept that he likes to incorporate into poetry that life should be a conversation between you and the world. And John McDonnell: like a really meaningful life or a great life is one where that conversation is really effective and goes both ways. And so that really reminds me of that. Jesse Han: Yeah, totally , it's a reminder to be open to what the world is telling [00:03:00] you. And I think that's really important to remember as you like go heads down and you try to make something happen in a startup. You have to be on the lookout for signals that maybe you should be doing something different. Maybe you should be pressing something harder. It's a careful balance that you have to strike. Research vs startup thinking --- Bryan Davis: Do you find that the signals you're listening to or the incentives present are different in a research context versus startups? And if so, how so? Jesse Han: To be honest, I don't really think they're that different. Like in research. So especially if you're in like a high pressure environment or if you're working in like a field that's moving really quickly, like AI , like what research looks like is taking a bunch of bets and choosing how to allocate your resources and figuring out Like what kinds of unfair advantages that you have that might make you unusually capable of capitalizing on the outcomes of some of those bets. And so I think that a lot of. So a lot of the thinking [00:04:00] around what kinds of bets one should take in their career apply equally well to startups and similarly thinking around what kinds of activities are useful for startups to think about, apply equally well to research. So like an example is pursuing like very high impact research events. Like you could spend a large majority of your career just pursuing incremental advances which carry less risk and are more likely to be published but which don't have an endearing legacy in terms of the research activity of others in the field. Or on the other hand, you can work on something that fundamentally changes the way that people think about some problem inside of the field. And that has a far more scalable... so I think a lot of the same thinking applies. Jesse's path from research to startups --- Bryan Davis: And how did you navigate with that perspective? You were previously a researcher, you were a PhD student, you recently finished your PhD, and you've obviously worked in technology prior to launching a startup. [00:05:00] But how did you, what was your own journey from going from the research context into deciding to work in industry? Did you ever aspire to be a professor? Jesse Han: I did at some point. So at some point I was very deeply enmeshed in the pure mathematics world. I was trained a logician for most of my undergraduate years. And then I spent my masters just studying mathematical logic and model theory. But I think that gradually shifted towards a more ambitious vision, which formed the basis for the research program which I pursued in my PhD. Partially due to my realization that I probably didn't have what it was gonna take to become a top mathematics researcher. I simply didn't have the let's say the intellectual horsepower. Because there are a lot of very talented people working in math, and it's a super small field. So to like really get up there, it's like being a star athlete. It's like you have to train every day, you have to study the work of the masters.[00:06:00] You have to be in the right place at the right time with the right advisor, working on the exact right field to me making that kind of impact. And towards the beginning of my PhD I came to the realization. The more impactful thing for me to do would be to try to just automate all of mathematics instead. And so I had this grand vision of eventually building some kind of planetary scale system for automatically searching for mathematical theorem improves. So that one day human mathematicians would just be the operators of such a machine whose details. And intricacies would be hidden from them, like an operating system hides most of its complexities from the end user. And so that was what sort of drew me towards AI and got me into more industry adjacent things because building a system like that requires a lot of engineering skill, requires some pretty compute heavy resource. And that kind of brought me into the orbit of people trying to apply the [00:07:00] latest techniques and deep learning to automate theorem proving. Automating mathematics --- Bryan Davis: To anchor a little bit more in the math world, do you ever think we'll reach a point in mathematics or perhaps are we already there where we're at the limits of the capacity for human brains to comprehend and do you think that there's a zone in mathematics, in pure math where machines will begin to inspire, be the chief creators of new algorithms, new theories, new results? Jesse Han: Yeah, I think that's a really interesting question. I think the fields where computers have a large advantage is with like really concrete kinds of combinatorics. That's one thing that stands out. Like subfields of discreet mathematics, like places where computation is really the main way to see how phenomena occur. For example, if you are like studying the dynamics of Conway's game of life, [00:08:00] then running computer simulations, or say you're just like studying say cellular automata, then running computer simulations is probably the best way to gain a good understanding of what's going on with any of the phenomena happening there. But on the other hand, if you're working in more abstract fields that require like a large tower of definitions say algebraic geometry, then. The computer based foundations get a bit more shaky because there are many ways that you can represent various things. And there hasn't been a lot of work on shoring up, commonly accepted foundations. Does that answer the question? Bryan Davis: Yeah, I think it does. I remember reading, I believe it was a, or listening to an interview with Richard Fineman several years ago where he was talking about understanding the universe as peeling layers off an onion, and his hypothesis was that, there may never be an end to the layers. We could just keep peeling and keep peeling, and eventually we might reach a boundary at which our capacity just to [00:09:00] abstract, our capacity to represent what is actually beyond the next layer, is just somehow limited or contained by the limits of, biologically based IQ or biologically based intelligence and I thought that was an interesting concept that I was, I'm curious to, to investigate whether the same thing might apply or whether you think the same thing might apply to mathematics. Jesse Han: Oh, yeah. There are totally trivial cases, right? Like there, there are like there are prime numbers that require more bits to represent than is representable inside of the human brain? Like that would be like a really trivial example. But you can already see this happening with the social fabric of mathematics. So what happens now is that a professional mathematician will go really deep and it's like harder and harder to become a true polymath. Like someone who's achieved mastery of like many fields of mathematics. And so what happens is that the social fabric of like mathematics is made up of these experts who only see a [00:10:00] very narrow slice of the entire picture. Like for example, there's a vanishingly small number of people who have a complete understanding of the classification of finite groups. And that's simply one piece of mathematical lore, which has been written down in relatively, maybe not low fidelity, but written down in questionable fidelity in a constellation of like papers and preprints and surveys. But the true understanding of the proof, the the thing that is communicated from a master to a student in mathematical practice is very hard to grasp. And it's only owned by a very small number of people today. So that's definitely happening and there are more examples in other fields of mathematics as well. This kind of phenomenon where where, you know it's becoming increasingly unclear. What parts of mathematics stand on firm foundations and what parts don't spurred a lot of research activity over the past few years in formalizing mathematics in a computer understandable way. This [00:11:00] research program was championed by Kevin Buzzard at Imperial College, where he drove a lot of people to organize a lot of mathematics in a computer understandable format, in a theorem proving language called Lean theory. And he gave a very good talk at Microsoft Research titled The End of Mathematics where he talks about things like this where there are so many parts of math where the highest standard of proof is just social understanding between mathematicians. And when you think about it, these things are on shakier foundations than you might first believe. John McDonnell: So where does that put us with, so you were saying that your hope was to build an, some kind of AI or automated system that could move the field forward essentially. And I think you mentioned like the foundations are firmer in a place like combinatorics. Do you feel like this is already having an impact? Or what are the kind of milestones to having impact. Jesse Han: Do you mean the milestones to having impact in terms of in terms of [00:12:00] like fully automating a part of mathematics or just in verifying the existing knowledge? John McDonnell: Maybe your perspective on both of those. How far is your vision from being realized even in a small way and what would it take to get there? Jesse Han: I don't think a system like this has really been constructed for any particular field of mathematics. Of course mathematics is vast and there are many talented people working in it. Like many of whom have, are, have been schooled in the ways of formal proof. And so a system like this might have been built, but as far as I know, like nobody's built like this automated proof search thing where ... So the thing that, that I would like to be automated there is like how mathematics research is conducted by a very senior researcher, right? Like they, they have this deep, deep understanding of the field and like what things are provable, what things should be proved, what kind of like [00:13:00] research programs should be carried out. It's kinda like building of like giant building, right? Like you say oh you can add an arch there if you like, use these tools from over here and because you have five years of experience already, it should only take you three months. And so that's like something which requires really intense focus, incredible amounts of persistence, superhuman willpower at times. And if computers were able to do that, and we were able to scale up the amount of compute that we threw at these problems. Then a lot of this could be automated and parallized, right? So ideally a mathematician could just come in and point to some location in the distance and say we should go there. And then an army of these AI mathematicians would go and do all the work that's needed for actually building that super structure that's needed for getting all the way over there. So I would say it's a few years away, but there are some pretty talented people working on this problem. For example there's the N to formal group at Google Research, which has been doing a lot of cool work in enabling [00:14:00] all the fundamental technology needed for building a system like this. They've done a lot of great work in auto formalization, i e the ability to automatically translate from natural language into computer form mathematics. Representing formal logic in language models --- Bryan Davis: I'm very curious to pause and focus there because a lot of the technology at Multi and obviously at OpenAI is about using natural language as the primary tool of interface with computers, but as very widely known at this point, their ability to do formal reasoning and complicated mathematics is limited. And there's a lot of things a lot of ways these things can blow up and fail. I'm curious, what do you see as being the key to weaving formal reasoning and formal logic systems together with large scale natural language models. Jesse Han: Yeah so at Multi, our mission is to automate all knowledge work and one way in which we envision that happening is by providing developers with [00:15:00] a series of trusted software components built on top of large language model primitives that have some kind of structural or semantic guarantee on their outputs. And so when you have building blocks like that you can trust that the kind of powerful occasionally unreliable intelligence that you get from prompt engineering, a pre-trained language model. So when you have those kinds of guarantees, then you can really begin to build sophisticated software applications that really tackle valuable real world use cases. And in that way, the massive amounts of compute that we have available today can be applied to create incredible value by being able to trust what these models are doing at a more granular level. Bryan Davis: How do you create those guarantees? That seems like the big problem. Jesse Han: So if you [00:16:00] look at the special case of mathematics, right? So what happens when you're using a language model to prove a theorem in some kind of formal system? So what you're really doing in that case is that you're synthesizing a program in a programming language, which might be implicit or explicit, that has a sufficiently expressive type system for capturing mathematical theorems. And so when that program is synthesized, it's done in a way that you can guarantee that program satisfies the specification, and that's checked by a trusted component of your theorem proving system. So if you take that kind of technique and you zoom out and you apply it to the more general case of building robust trustable software on top of large language model [00:17:00] primitives. Then how that looks is you want to synthesize code, you wanna synthesize entire data structures, which are subject to some kind of specification that is computer checkable. So what we're doing is like very related to some recent work that people in the AI community have been doing structured extraction and the creation of knowledge graphs from informal text. But we wanna provide even stronger guarantees. We wanna provide more modular components that people can use not just as, things to enter into a SQL database or things to add into, like a knowledge graph, data structure, but components that people can build software with, software that can plug into the world and take actions. Software that can like power the backend for a far more complex application than just a chatbot. Solutions to producing verifiable systems --- Bryan Davis: One strategy I've heard to resolve these issues is basically [00:18:00] a trial and error method whereby you ask a model to come up with a solution to a problem that sort of satisfies some condition , and then you check the condition and if it fails, then you just ask it to regenerate. Is there any other secret sauce that can resolve these issues, that can make these things better or more reactive to the feedback from a checker? Jesse Han: Yeah, I have a couple tricks with my sleeve. You'll see . But so you can always just add a function that calculates a bool at the end, right? . So then you can be like, with max retries is equal 16, sample me something that matches this model. And you pray that everything actually parses. That's obviously not great. But if you, so if you approach it the right way with all the expertise that like one might get from trying to solve Olympia math problems in a formal theory proving system, then you can get really far and you can begin to see some really cool things. Like you begin to do what feels like the future of programming, [00:19:00] right? Where where not only do you have this black box that's able to like, transform strings into strings, right? This like programmable fuzzy black box that represents some, some distribution that you're just sampling over all strings But you begin to write programs that, that reliably use language model primitives to achieve some goal that would be impossible with normal programming alone. So we're gonna be releasing this in a few weeks. People are going to be able to access capabilities that Multi has built on top of language model primitives through an api, they're gonna be able to integrate it into their own AI applications and the kinds of guarantees that we provide make it possible for people to build AI applications that can rely on certain kinds of outputs, right? Like you can process an entire document. You can make sure that you're [00:20:00] extracting text that always support certain claims which are made. You can create so you can create data structures that are constantly accumulating like a stream of thought, a chain of thought audit trail of like why the model is doing what it's doing. And that kind of interpretability, that kind of transparency into what's happening under the hood, and that kind of sophistication is something which is only unlocked when you have a library of trusted and modular software components. So that's something that, that we were building towards with Multi Flow, which was the first product that we released. So that's like this low code workflow builder. That prioritizes a bunch of AI capabilities. But really what we want to do is we want to expose this to everyone and not necessarily have those workflows only accessible through our low-code builder. We believe that all developers who want to build these kinds of new AI applications, anyone who wants to build their, their own internal version [00:21:00] of chat, p t plugged into their company's internet should have these kinds of reliable software components so that they can build as confidently and as rapidly as possible. Vision for Multi --- John McDonnell: Maybe it's actually worth because we're alluding to it I'd be curious how you would summarize essentially what Multi is and at the big picture, what the vision is for Multi. Jesse Han: Yeah. So we're going to automate all knowledge work , and how that's gonna happen is that we're gonna provide people with these building blocks that represent, at first relatively basic units of automation, but which will over time become more and more sophisticated units of automation in a verifiable and trusted way. And as people continue to use Multi components in their AI applications, they'll be able to build more rapidly and more confidently. And, eventually there's going to be capabilities that we ship that are going to do [00:22:00] tasks that were once thought to be entirely under the purview of humans. Something as complex as taking a quarterly financial report from a Fortune 500 company and compiling it down to a compressed report along with a bunch of data that's entered into a spreadsheet that kind of like structured extraction, critical thinking, reading comprehension, something that like might take someone who's working at the lower rungs of, say, an investment bank, several hours to do will be compressed down to several minutes by using software that has been built out of the components that Multi provides. Alignment and verifiability for ambiguous goals --- John McDonnell: Yeah, and one thing I was curious about in terms of this idea of having these components be just be very reliable at providing the desired kind of outputs, is that there's things that you can instantly check in an automated way. So if I'm generating code, you can just immediately tell me if my python code is syntactic. And so I [00:23:00] can see how you can just really verify, okay this thing's gonna output Python code. And I'm gonna one thing I'm gonna verify is that it is, that it's syntactic. It seems a lot harder to do that for concepts that are less simple to define for a computer. And so just like an immediate example would be for a financial report. If I'm doing, if I'm trying to read an FTC filing, and I'm gonna pull out of that certain facts about the company, there's a lot of complexity, for example, with how accounting works, and so there are things that seem really fuzzy to me about exactly what maps from what's in that report to what you're gonna want to have in the spreadsheet. And it's, it feels to me really hard to automate the checking of the accuracy of the output. Jesse Han: Yeah, totally. No, so you've hit the problem right on the head. Which is that because we don't have a perfect formal representation of all the concepts that we want, right, there's no principia mathematical that we can write here. There's no formal logic that we can write down that specifies that 'oh, this [00:24:00] is a good this is a good summary of this SEC filing,' right? This is what your manager at the investment bank wants. So because we don't have a kind of algorithmic check, we don't have a way to statically analyze the algorithm that produces those outputs and to guarantee that they actually satisfy that specification. There's this problem of you give instructions a natural language. How do you ensure that the outputs are actually aligned with those instructions? Because if the instructions are a natural language and the specification is a natural language then the only way that you can verify it is by having some other agent that understands natural language look at those outputs and hopefully you find a way to, to minimize the likelihood that it's not aligned. John McDonnell: But there will be a lot of context that's not in the language, right, so it's like my boss at the iBank says summarize this FTC report for me. The instructions were to summarize this report. Like I know a lot about my boss. Like I know I have a lot of contacts for this situation. [00:25:00] Like I, I have a lot of ability to fill in a lot of gaps about what's really needed that are not, that are just like absent from the stated instructions. Jesse Han: Yeah. So I think that the way forward there is that you have to break down. So you have to break down the the task of whatever kind of discriminator you're applying to your inputs and outputs in such a way that you can, so in such a way that you can apply large amounts of compute through language models, because what's gonna happen is that, so over the next few years software, like the kind of software that I'm describing, this kind of neuro symbolic software is going to be responsible for an increasingly large fraction of economically valuable activity. And in order to provide guarantees about the correctness and the alignment of the actions and outputs produced by that software, we are going to have to apply the same kinds of [00:26:00] techniques used to build that software in order to provide bootstrapped guarantees about that software as well. So I don't think there's a way to completely guarantee the correctness of what comes out of that kind of software. Except for having humans check the outputs. But I do think that what we can do is that, that we can apply the same kinds of language model based compute to to provide soft guarantees. So we provide enough compute and we apply it in the right way, then we can push the probability that the outputs or actions are severely misaligned way out, to the point where if there's like a 99.5% likelihood that it satisfies like the the extremely convoluted say judging process of these very these five very large language models then it probably won't [00:27:00] need human supervision. And that's something which will benefit from a data flywheel effect as well. So as we get more and more data from people who will be judging the alignments, the alignment of the kinds of inputs and outputs that we see there. We'll be able to train better and better systems that will provide stronger guarantees, but we won't ever be able to completely get rid of the possibility that there is some extremely unlikely output that comes out, which is completely misaligned. John McDonnell: I feel like there's also, there's a bet here, right? Which is because one way, sometimes people think about this. This is oh, I'm gonna get my model to the point where, in validation, where like I do a one-off validation. So I do a training and then maybe I do like my fine tuning and then okay. It looks like I have some kind of validation set where the model's performing up say 98% accuracy or something on some metric I came up with. And so I, I could just say oh, I'm just gonna ship that. And I know, like as long as I stay within [00:28:00] distribution like I'm just gonna expect the model to continue to perform at that level. Versus like the alternative approach of saying actually every time I do inference, I'm gonna have a validation check on that inference. And I feel like one thing that, and I actually haven't heard people making this really strong case as many places. So it's like a really interesting contribution from the way you're describing it is that it's this bet that that doing online validation at inference time is gonna be really crucial. Is that accurate? Jesse Han: So I suppose it depends on how strong the underlying language model is. There is a language model or some ensemble of language models. Like, for example, you don't really have to use the same language model for everything, right? Like you can just use like super strong language models for the most cognitively taxing parts of some task that you're writing down in this knowledge work automation software. So it depends on how strong that software is originally. But to be [00:29:00] honest, my take is that , that kind of verification at runtime is just part of the execution of the program. I think applying, so I think applying language model compute that way is simply part of the toolkit that should be taken for granted when working in this new style of programming. And the only thing that we can ever do is we can just turn up the compute to push that tail risk further and further out. And I think that's something that's actually incredibly valuable, right? Because if it's a low stakes kind of economic activity, maybe you don't really care if the email spam that you're sending out for SEO is like not great, but if it's like a high touch oh, these, like 512 sales emails have to be really awesome so like we better... You just have to provide a way for people to signal how much they value the correctness of the outputs. If they want to pay for more compute if they want to pay for more intelligence then in a world where intelligence is too cheap to meter, that's not really a problem for the customer.[00:30:00] And in fact, it's actually a net positive. Bryan Davis: I'm curious to zoom out a little bit and talk about the trends of the industry. You've spent time at OpenAI, you were there building as a research scientist. Now OpenAI obviously they're doing this sort of machine learning as a service for the industry at large, and they're also partnering very closely with Microsoft. Where do you envision the provision of large language models and models themselves as an industry going? Do you think that there's going to be a lot of horizontal scaling? Will these things become more commoditized over time? And how does that relate to open source projects like LangChain? Jesse Han: So the question is How does the potential commoditization of language models have an impact on open source projects like LangChain? Commoditization of intelligence --- Bryan Davis: I'm curious about the kind of overall take on the, maybe the topology and the players in this space, whether or not horizontal sort of expansion of people who are building models and building sort of cornerstone models like OpenAI and like Anthropic and these others, and the trend there to basically [00:31:00] provide machine learning as a service, versus the sort of vertical specific approach of building a specific product. For one thing, I think it's interesting to that ChatGPT has really been the accelerator for a lot of OpenAI's success over the last three or four months, which is a very specific instantiation of their technology. And perhaps that'll divert them from a business perspective away from this more horizontal commoditized approach and more to a product specific approach. Curious what you think. Jesse Han: I obviously can't speak to precisely what they have in mind for their strategy. It's been a while since I've been there. But I do think that the world towards which we're headed is one where language models become increasingly commoditized. Compute is still a bit expensive right now. The total number of dollars that you have to pay per unit of intelligence, so to speak, is like still relatively high. And it's still a bit cost-prohibitive to run [00:32:00] purely magical feeling apps on top of AI. But I think that's definitely going to change. I think the world towards which we're moving is where... so where language models, speaking precisely , the forward passes of language models that are equivalent to say what GPT-4 will be. I think we're moving towards a world where that kind of intelligence is going to be as cheap as water, where the availability of language models will basically feel like the availability of microprocessors right now, right? Like right now we don't even think of microprocessors as obstacles to building, right? They're like these completely commoditized things. They sit in every smartphone. There, there are more microprocessors on earth than humans. And we're definitely moving towards a world where the same thing is going to be said of language models or their descendants. And so I think that what's going to be really important in that case is the kinds of software and infrastructure that you can [00:33:00] build on those primitives. Microprocessors weren't useful until. The various layers of firmware and then low level and high level software were built that could make them really accessible to end users. And I think that, similarly, a lot of the value from this existing AI wave is going to go to the people who go that last mile in building out the rest of that stack. And I think LangChain is a great example of a project that consolidates a lot of the knowledge that's needed to get builders going. LangChain and Multi --- Bryan Davis: Do you anticipate LangChain or projects like LangChain as being competitive with Multi or something that is integrated and part of Multi's journey? Jesse Han: Oh, I don't think they necessarily have to be competi. I think it's natural that there are gonna be various sorts of software frameworks, software providers and services providers built on top of a new platform, like the language model [00:34:00] platforms that we're seeing right now. We're definitely not competitive currently. The future --- Bryan Davis: What would you say is the one and five year vision for multi? Where do you want to take it? We've gone into it a little bit, but curious to think about what are you most excited about in the next one year, in the next five? Jesse Han: So I'd say the thing I'm most excited about for the next year is the increasing availability of language model technology. I'm really excited about some of the upcoming open source releases. And I think that once once more people have access to. This kind of commodified intelligence, the more demand that there is going to be for the applications built on top of that. And I think Multi's gonna be able to provide a lot of the tooling that people are gonna need for building really economically valuable applications. As for where we want to be in five years. So I really want multi to [00:35:00] ...to provide intelligent automation for every sector of the economy. Like multi should be the cognitive engine, which powers like, I don't know, 60% of what used to be investment bankers, 60% of what used to be the people who would compare the terms provided by insurance providers against like certain claims. Should be like 60% of the people who just digest a bunch of information from various data feeds and prepare them into reports at various market research firms. There are so many cognitively taxing tasks right now, which are performed predominantly by humans, which I think are right now perfectly within the range of the kind of automation that you can build with language models. The only thing that has to be provided is the infrastructure and offering of trusted components, so to speak, that lets people create that and roll it out at massive scale. I want to be in [00:36:00] a place in five years where it's clear that we are on a path to automating all knowledge work, and we have begun to automate a serious fraction of knowledge work and at least several important parts of the economy. Time at OpenAI --- Bryan Davis: I'm curious, what were your takeaways from your time at OpenAI? What did you leave there feeling you needed to be done. Obviously you journeyed from working at OpenAI to starting your own company. What did you feel like was missing from OpenAI that made you feel like there was something else you needed to. Jesse Han: That I feel was missing at OpenAI. So OpenAI is a very special place. It's got great talent density. It's got some really motivated and hardworking people. It's got a very unique mission. There, there aren't many places in the world that have a credible claim to being the potential locus of AGI with as much real world impact as they've been able to achieve. That being said, by virtue of the clarity and focus of their mission there are many paths [00:37:00] that they are unable to take because they're not on the critical path to AGI. And I think that the emergence of this technology created the opportunity for many trillion dollar companies. And while OpenAI is perhaps positioned to capture that value I think that it became increasingly apparent to me as I was working on this technology that there was room for many more. And I wanted to go out and create that kind of real world. , I want it to be on the front lines. Connecting that technology to the real world and going out and automating all knowledge work. That doesn't necessarily mean scaling up to the next gigantic cluster and building GPT-5. That doesn't necessarily mean pushing the latest state of the art in theorem proving with language models. I think what that means is doing the highest impact thing that I could for this vision [00:38:00] of trying to automate all knowledge work. And for me, that meant going out and doing Multi. John McDonnell: What important stuff did you learn at OpenAI that helps you carry forward to, to this next journey you're doing? Jesse Han: Things I learned? I mean I learned a lot about language models and those are sure important in our line of work. I had some really great mentors when I was there. I think something which I was particularly inspired by was, was the strength of Ilya's conviction in whatever research program that he was championing at the time. He really taught me the value of really strong conviction, held in the right idea, compounded over a long time. And while that doesn't sound like much, I think it's, so I think that kind of intellectual courage is really needed when you're on the forefront of something [00:39:00] like this. Because, so especially in a field like deep learning and concomitantly in the field of building startups in generative AI, there's so much happening around you at all times, there's so much flux. There are always like new papers being released, new like flashy trends being pushed by various people trying to get their careers off the ground. So many like flashy releases on Twitter that it's really important to keep your eye on the ball and to remember the principles. From which you came to your conclusions about why you should be doing what you're doing and having the courage to pick something and to just go at it in spite of all naysayers, in spite of all setbacks, I think is really valuable. Advice to others --- Bryan Davis: I'm curious what your advice would be to those, perhaps new arrivals to this space, [00:40:00] either in the academia or people who are suddenly tuned in to the ChatGPT craze, people that want to use these technologies in creating new businesses and solving new problems. What would you, what would your advice be to people who want to shift their career in this direction? Jesse Han: Well, there's all the general advice, which is of course, work really hard, try to find some really good mentors, people who can teach you a lot. Find really ambitious people. Try to work as closely with them as you can. In terms of like people starting companies right now, I would say again, general device, like you should swing for the fences. Cause it's so easy to pick something that's like hot. Something that might be flashy for a month or two. But I think so I think aiming for something that really endures is.... I think it's hard cuz you have to think deeply about what to work on [00:41:00] and you have to really believe in the thing that you end up doing. Yeah, if I had to sum it up my advice would be swing for the fences in every sense of the. Bryan Davis: Love it. John McDonnell: Yeah. It really comes across that you're doing that. Like you're gonna automate all, all knowledge work. That's like such a legit, extremely ambitious vision. Bryan Davis: Yeah. Moonshot. Recommendation --- Bryan Davis: Jesse, we love to end our interviews with recommendations specifically on something to read, watch. Is there any sort of media it could be anything from a song, a poem, a mathematical theorem that you'd recommend that our listeners sort of tune into? Jesse Han: Song, a poem, a mathematical theorem... one book that, that I could really recommend is... in general, I think it's really important to remember that, it's important to remember that the dawn of computing is still in living memory, which, if you think about it, is really [00:42:00] remarkable. There are people alive today who, when they were born, you know like the Soviets were like still trying to assemble like room size computers from like glass tubes. And there was no way for this person who was like born in a Siberian Tundra to even know that something like language models would be down the road in like a sizable fraction of a century later. One book that I found really interesting about the early history of computing and software is Steve Lohr's Go To. So there's an interesting account in there about how Ken Thompson and Dennis Richie wrote Unix in three days, cuz they were really like, Fed up with the Multics operating system and they thought why don't we just build something better with like composable components that you can pipe to each other. So I feel like, [00:43:00] so I feel like there are moments of history where like time and technology really align and the veil grows thin, so to speak, and really talented people working together, on the right problem at the right time, can push through the curtain and build things that have incredible outsized impact. People like Ken Thompson and Dennis Richie can build Unix in three days. And forevermore influence the history of computing. And then they can reco collaborate later on and build the B programming language and then have another crazy outsized impact on computing. And I think that so I think that we're really going through a similar period right now. So I think that with this current wave of technology, the veil is again thin, and sufficiently talented people working [00:44:00] together, focusing on the right problem, can really go out and build something that might change the world. So I think it's really inspiring to think about how in previous previous cycles of history, similar alignments of time and technology have enabled crazy amounts of change. And I think the dawn of computing is a great example. So the dawn of computing and software specifically. So Steve Lohr's Go To great book. Bryan Davis: Thanks, Jesse. John McDonnell: Yeah, I'll check it out. Thanks so much, Jesse. Bryan Davis: Thanks for being a part of Pioneer Park. Jesse Han: Yeah, thanks for having me. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit pioneerpark.substack.com (https://pioneerpark.substack.com?utm_medium=podcast&utm_campaign=CTA_1) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

9. 

   1 May 2023

   AI avatars and creator alignment, with Avi Fein

   Avi Fein, founder of Meebo, discusses how AI can be used to extend people's capabilities rather than replace them. He explains the differences between Meebo and ChatGPT, and how YouTube's success is due to its product definition and monetization engine. He also talks about the importance of trusting individuals rather than brands when it comes to moderating the internet, and the road to monetization. A great and wide-reaching conversation. Transcript John McDonnell: Okay, so we have with us today, Avi Fein. Avi is the founder of Meebo, which is a platform for building personalized chatbots. Prior to that, he was a member at South Park Commons, and previously worked at Neeva and YouTube and Google. Avi, welcome to Pioneer Park. Avi Fein: Thank you. Great to be here. Bryan Davis: Yeah. Welcome. Good to see you. So we've been having some conversations prior to this, which I think at some point we all realized oh, we should probably turn on the microphones just so we can begin to capture some of this. And I think we were just on the con topic of talking about how to master chat and really like [00:01:00] some of the challenges of chat. So first, can you just tell us a little bit about Meebo? Avi Fein: Sure. So Meebo is a platform where we build chat bots out of creators of various topics. We look for people who are usually like experts in a certain thing and, really have proactively shared their knowledge. And then on the other side, there's people who trust them and want to connect with them to get almost like one-on-one advice for recommendations for, questions that they may have. Where so much we're going to. I would say Instagram and TikTok and YouTube nowadays as being the place that we want to get knowledge out of and we want to get information from. But those are still static and a distant in many ways where, they're not relatable to you. They, can answer, really connect with things that you're interested in, and we want to break down those barriers and really use chat as an interface to make it interactable such that you can have conversation and go into the depths of both you and how it connects to that person and their knowledge and their content as well. Bryan Davis: Cool. So I guess something that's really top of mind, I think for a lot of people right now is ChatGPT differentiate. Tell us [00:02:00] how Meebo is different than just run of the mill vanilla ChatGPT Avi Fein: yeah. It's interesting cuz we started working on this before ChatGPT even came out, but I Very hipster of you. Yeah. But I would say the foundational ideas and principles actually cut across. Even the post ChatGPT world and that one, what we wanted to do was break apart knowledge to not have it be a monolith anymore. And if you looked at what a lot of people experienced with the web and the internet today through products like Google and now ChatGPT it's relatively generic. You get the same answer independent of who you are. Like if you do a Google search, if you do a ChatGPT like Q and A, we're all gonna get the same thing back. And our belief is that it's a much more. Delightful. And not only that, but like trustful experience when you can blow that up and go into the distribution of different perspectives and different niches of knowledge where someone's gonna have a slightly different take than, person A versus person B on a whole slew of things. And so for us it's how do you take, [00:03:00] some of the technology that ChatGPT is good, but apply it to the diversity of human perspectives and knowledge in many ways. I think the second part that we build on, That's beyond ChatGPT is playing with the idea of how do you use the technology to extend people versus replace them. And a lot of what people talk about in AI now is like these virtual assistants, which are just. SNTs of like humans where it's oh yeah, we've trained on a million of you and now this can do what all million of you can do. Like you should just use this one like AI bot. And that's true for art. It's true for now chatGPT and knowledge of like why would you talk to anyone else about ChatGPT it knows the entire internet? And I think what goes unsaid in those things is that when you do that, you lose the integrity of in the nuance of all those individual people and all the individual relationships and the trust even that you may have in that. And it becomes not to go back to the same [00:04:00] idea, but there's like monolith of just, the average across everything. And what we wanna lean into is, The individual. And it is like the personal, and it is the idea that we are all unique in our way and like how can it, how can AI extend us to give us superpowers versus just act as like a replacement of us all? John McDonnell: So when you talk about that uniqueness, certainly from a in the first comment you were saying, oh unlike ChatGPT, we wanna be really personalized. How are you able to achieve that? Avi Fein: I think it starts with people, and I like, we said before of what Meebo is like our building block, our atomic unit, was an individual creator, was as someone on YouTube. And really it's actually cutting crosses. It's like the person has, they're represented on YouTube, TikTok, Instagram, and even their website like that is like your identity. And so we started with the identity as being the atomic unit and then build up from that and the, Philosophy behind that was that you can capture their unique [00:05:00] perspective and their, you're their unique point of view, and then make that accessible and shareable with the world. And by doing that, you can maintain this like boundary so that it's no longer like the aggregation of them, plus 10 others who are like them. And then you actually lose texture and you lose the nuances of like their experience of the world. And you also then from the other side as a user, Know who you're talking with and you can have a trusted relationship versus being like, you have to take this leap of faith with ChatGPT, though what they're saying to you is the authoritative truth of the internet, and you're like, we're in a post truth world. What is the truth of the internet? Bryan Davis: Yeah. It brings to attention some of the interesting issues. A lot of the complaints about ChatGPT and related products have been that they hallucinate that the things that they spout so confidently aren't facts. Which I think has been a warning sign for a lot of people. But it is also true that the perspective of an individual creator is also not necessarily a fact. So I'm curious to hear your perspective on two angles. One is the ability to take a creator's perspective and actually [00:06:00] represent that faithfully. Like how do you ground your technology in the actual perspective of creators and how do you feel about creators being obligated to be truthful? For instance, fake news. Like what's the, what are the risks or maybe down the line for how Meebo could be a voice of people who you don't necessarily want to expand their voice. Avi Fein: Yeah. I'll go in reverse order because I think the first question is almost like the harder question than the second one, at least for us. On the second one, and this connects with the idea of like, how do you not think of the world as being a generic monolith of information that we all trust in that we're not trying to give you an opinion around what is fact and what is not fact in the world. By virtue of talking with an individual, you are establishing that you trust them, or at least like you like, like they're their source of knowledge and they're the source of information, not us. And having worked in this space before, at least at like Neeva and seeing some of these [00:07:00] dynamics and that one of the drawbacks of the, of those types of products is that trust is inferred into the brand such that I trust the first result on Google because Google said it's the first result. And the actual sourcing of the individual things that like go into it, start to fade away and not matter anymore. And then Google becomes responsible for moderating the internet. And then Twitter becomes responsible for moderating Twitter. And then Instagram becomes responsible for moderating those things because trust flows up into the brand versus like staying down into the individual. And they all say oh, that's what we, we don't want to have this responsibility, but they design the products and they build the products to, because they like, become the aggregation point and become the, they come the center point to do that. And for us I think it is about not meddling too much into those worlds and letting the individual points of view, the individual facts still sit where they like lay. Like we're not gonna strip it out like of someone's chatbot just because we may disagree with it. Because you on the other side are an [00:08:00] adult and like we trust that you will be able to form your own point of view on whether you can have that trust with that person or not. And that's the complexity of life and I think the reality of it. On the first part of how do you actually do a good job of this? That's like the long arc of technology and I don't want to claim that like after three or four months we've solved some massive problem and be like, ah, guys, like this is done. Like we we've done it here. What I can say that we lean into and we think give us tailwinds to be able to tackle this number one is that we come from a background in search. And what that means is that we spend a lot more time and energy and effort into. Retrieval as being like an important problem and understanding what are the facts or what are the opinions, or what are the things that this person has said and how do we like, make sure we're relying on that. And what that does is it gives you a boundary in terms of the AI and what it like when you are generating a response or when you are trying to [00:09:00] Yeah, like generate like leverage that you can know apriori how far you may be deviating from what that person has said before. So imagine like when it comes in, you have now GPT3 where you give it a prompt, and when you're assembling that prompt, you could say this person has Neevaer said anything about this topic before. Or like they've said something, 20% about this topic, 80% about this topic, 70% about this topic. You can at least now have like thresholds to say they have not talked enough about this to really give a response and just be like, I'm sorry I can't answer your question. For example, and that's like a very simplistic I think approach to it. But in concept, I think once you have a source domain that is like boundary incense on it you can apply some of these techniques to. Not let the model hallucinate or at least know when you're hallucinating more. John McDonnell: I do wanna double click on something here around these recommendations because you worked previously at Google, YouTube, and Neeva. And one thing that I've always found so these platforms do end up becoming editorial and I've actually always found that YouTube seems to have the [00:10:00] most wholesome recommendations relative to other platforms. I find that like when I get recommended things on YouTube, they're often educational or interesting and relevant to my interests, but not in a perverse way. Whereas, say TikTok is clearly just trying to addict me to their platform which is fun, but doesn't necessarily feel as wholesome. Why is that? Bryan Davis: Or perhaps how could that differentiated experience be created? Avi Fein: Yeah, no I love this question and I love YouTube And I feel the same way. I think that's like definitely a very true observation with insight into it. If I had to speculate on the potential reasons for it, I would guess it relates to both the product definition of long form video itself. And then also monetization and how those relate to why this, like maybe manifestation of it. One is that I think more nutritious edu educational content is hard to make bite size. And it is better in like a long form format. And I would also say, I would also guess intuitively that the people who want that type of [00:11:00] content don't want it to be like overly reductive and to be like this like hot take TikTok type of thing, where it's no, they're interested in going deep and actually like learning about this thing which is not well suited to those short form bite size types of. Like platforms. And so I think there's just a natural product definition that causes more of those things to flow into a YouTube or it's a better fit for both the audience who like want to engage with it. And for the content itself in many ways. I think the second reason for monetization is that outside of YouTube, it's actually very hard to make a living generating content on TikTok or Instagram. They're just not the same type of monetization engine for creators as like YouTube is. That's largely related to both. YouTube's an amazing monetization engine of like they can Make billions and billions of dollars of advertising. But they also share all of that with these creators. And if you take the power of Google's advertising machine and then you [00:12:00] give 50% of that, not exactly, but roughly 50% of that out to creators. You're sharing a lot of wealth and they have, they show more wealth with creators than any other platform by far. What that also means is, Some of these things which are less popular and less like mainstream, like educational informational things are, can survive and make a living on YouTube where they really can On TikTok or Instagram, if you are like an info entertainer, nutritious content creator you really aren't gonna make a living on TikTok or Instagram. You'll probably do it for a few months and then realize how hard it is and how much it's just like running up Mount Everest effectively and probably burn out and not do it, and fade away. We're on YouTube. You can find your niche and you can find that audience because of the platform, be, and then be able to make money that comes back to you on it to reinvest and actually have a content business that like comes out of it. Good. Bryan Davis: What a, what allows for YouTube to be that platform in a way that Instagram is not. Both of them have enormously high user counts. I would imagine that the kind of number of crevices and interests that one can [00:13:00] fall into are just as deep on all of these platforms. What do you think makes YouTube distinct? Avi Fein: In terms of its monetization, you mean? Bryan Davis: Or if it's, I guess its ability. Yeah. One thing, it sounds like one of the biggest differentiators is the ability to make a living. Yeah. What incentivizes YouTube to keep that open? Because it sounds like if they wanted to squeeze artists, they could, but they're choosing not to. Is that motivated in the core leadership of YouTube, or is that something else? Avi Fein: Oh, ab absolutely. There's YouTube caught onto this far earlier. To me than other products did in that content creators are your supply of unique, differentiated content and also then the relationships that people are coming back for like more and more onto it. And if you want to have. A healthy marketplace, which these products are ultimately marketplaces, where like content creators are coming in as your supply and you users are there to consume it. You need to make sure that your suppliers are able to[00:14:00] reinvest into their businesses to make it better and are able to increase the quality of their content and their output over time. You don't want to actually squeeze your suppliers so much that you're effectively selling like trinkets that are, low value, just like commodities like. Crap effectively versus able to build up into the value chain and offer better and better things. And I think there is a deep insight that YouTube had there of we want to make sure that our supply gets better over time. We wanna offer a higher and higher quality product, which means we can't squeeze margin out of our content creators and keep it all for ourselves. Because what you'll do is you'll undermine yourself in the long term. And I think that was in many ways an early insight of the partnership program and it's something that has allowed it to thrive and and achieved a lot of these goals. Bryan Davis: A good note for us, John. Yeah, very much yeah. John McDonnell: How do you think this is gonna end up playing out for Meebo? Avi Fein: The idea of like revenue or monetization. Yeah. Or the like, How do the dynamics of like education or like advice or [00:15:00] information then connect into Meebo, like chatbots? John McDonnell: I think, you listed out essentially two reasons why YouTube might have more kind of nutritious recommendations. And I think the first one was a medium is the message type of thing. Yeah. Oh, just YouTube medium is just suited to high investment content. Yeah. And then the second one was around essentially the incentives of the creators. Yeah. So as you. As Meebo expands out, what, what's gonna be the character of Meebo and how are those two elements gonna play out? Avi Fein: Yeah, I think it's no surprise, that when we talk, when I talked about before around, where we're starting is in the more of like expert and advice area. And the reason is that when you say medium is the message, right? Yeah. And that we got really excited by. Chat bots and chat agent conversations in this context because it allows you to go into the nuances and into the one-on-one and the personal. And that is like where the medium shines of chat. And I would actually like then, juxtapose this against ChatGPT, where it does really bad.[00:16:00] This has a chatbot. It's actually not an if you like, were to compare a conversation you would have with someone where you were, give it the same input or the same prompt as you give to ChatGPT. That's not how a person would respond to you. No it, it it's not, it's a chatbot only in the sense that you can have follow-ups that like iteratively build on it to have this idea of like memory, which Google does like is doing in the background but doesn't, explicitly do in the product interface. But I think what is really exciting about chat and about dialogue, which we were talking before, is in the back and forth, and it is in the nuance, in the details to say " Hey Bryan tell me specifically like what you've learned here, what you've done so far, and how can I apply my specific knowledge into that?" And to me the medium of chat is very much. Like about you both working together to strive to those goals or, and it could even be just getting to know each other better as well. I think on monetization, we're a typical startup of build an engagement. And make build a successful, engaging thing first, and then monetize [00:17:00] it. I'd say in the spirit of the product and of the company and the spirit of me, if we can monetize, oh yeah, we're sharing that with creators, we're sharing that with people on the other end because the exact same reasons that I said, and just in the fundamental values that these are the people who are the authors, are the creators and the ones who are the thriving suppliers of the product. Like to me, like I see them as like the blood of these products. And we haven't talked about Neeva yet, but even one of the reasons I went to Neeva was feeling that products like Google don't value the web and they don't value the, like the free publishing that everyone gives them as a supply of the web, and they are extractive at the end of the day and they act as like a gatekeeper in a toll really for discovery and for connecting with those people. And it just was contrary to what, to me, felt like sustainable, equitable ecosystems that have like fairness and have the idea of, rewarding people for good content built into them. Bryan Davis: We're now going through [00:18:00] this massive wave of interest in generative ai. It's a huge sort of hype cycle. We're talking about products that can be built off of this and products that can be based on this, basically grounded chat that represents soMeebody. Where do you think this current wave is overhyped? Where is it over promising and where do you think there's potential to where it's oh, there's an unexplored direction in this way? Avi Fein: You mean specifically within chat? Within like Bryan Davis: specifically within, let's say, generative text. Avi Fein: I would say it's most at risk of being over-hyped in the deep domain use cases, and we were talking about those before. I think there is a risk that it's overhyped in like B2B where everyone wants to flock into, because typically B2B businesses are like money making things that investors love and other people love and stuff like that. The risks there are, the expectations of B2B are really into nuance and getting down [00:19:00] into a deep domain depth of someone's individual business. And a lot of things that are hard to capture in the digital world. Even of oh, Lucy talked to Mike about z like, and how does this AI bot not know about that? And you're like there's not really a good record of that anywhere. And if there is, it's like very partial and hard to even connect with what you're expecting out of it. And I would say in the workplace, people I think will have different expectations and those ex like of these AI agents and those expectations will be hard to live up to given the information available. And I think the difficulty in accessing even all of that information in different warehouses and places that it lives in unstructured ways, in the messy world that like a business is. And I think that's actually less true in consumer. I think that you have far lower consumer expectations actually, of these things and like what they can do. And you have far more information available that will help people with the 80 or 90% of the use cases that [00:20:00] they will actually have in practice. So it's more well set up. I think, than people probably give it credit for where do I, where I would say where things are under hyped. I guess consumer I touched on that one a little bit. John McDonnell: Although ChatGPT is consumer. Avi Fein: What? Yeah. Yeah. ChatGPT is everything, yeah. Yeah. I wouldn't really, yeah, I guess it's is consumer, but it's like you've seen it most in like the B2B use cases, take off. I feel like it's, that's like my impression of it, of like people see it, of applied at their job of or applied as a student, which is almost like a b2B use case if you look at it in many ways. Yeah. That's interesting. I don't think you're going to chat GBT to ask it what to wear out to a party tonight or something like that, or like what to do this weekend or those classic like consumery types of things. John McDonnell: I have used it to plan a party. Avi Fein: Yeah. Oh yeah, you did. Like you did have, yeah. I'd be curious on how it worked for that. John McDonnell: I think ChatGPT is very good at generating generic checklists. Avi Fein: Yeah. And so you got a checklist of what to do for your party. John McDonnell: And that sounds trivial, but it's really not in some way. I think, there's the Atul [00:21:00] Gawande checklist manifesto, et cetera. When you're doing something moderately complicated. It can be easy to miss something important. And if you just have the source that can just generate your checklist, then maybe you would've just generated the same checklist on your own, but you can double check it against theirs and you can just say oh, did I miss anything? Bryan Davis: Yeah, a generative work. Like I can imagine having it generate a workout for me and like that being something like, oh, okay, that's reasonable. I had a good goal for the day. Avi Fein: So I think that but those are such good places to juxtapose against because I. Yeah. The magic to me that's like super exciting to work towards is you're planning a party and you're like, I want this party to be special. I want something different. I want like an idea that is not the generic. I don't know that ChatGPT will be able to get that to you and a friend will. Yeah. Like I, I think like ChatGPT, if you say Hey, ChatGPT, gimme like, special ideas for my party. I still think it'll like, spit out something to you, but I think you'll still kinda be like, eh, I don't know, none of those really hit the mark with me. And same thing for like an exercise routine. Like it'll give you an exercise routine. My guess is you're on the [00:22:00] other side being this doesn't really hit the mark with me. And I, and there's a, there's a. There's a, there's something to bridge that gap there. And I think AI can bridge that gap. And I think like we have the capability to do it. And I think that's there's like an under hyped goodness. And I can't articulate or even know exactly what there is there, but I'm like, I know the tech is there and I know the information is there. And that bridge, I think is through ... Bryan Davis: retrieval. Avi Fein: It's through retrieval, but it's through like it's through people and it's through nuance and it's through the idea of like, how do I get to know you, Bryan and you John, and like, how do I use like the unique things about you and the unique things about the people that you like to talk with to give you something more special for you? John McDonnell: It is interesting. So I feel like say MidJourney has done a good job making their platform produce art that people are just more into. Do you feel like that's just, oh, they just exercise editorial control and that maybe there'll be like a dozen different mid journeys with different aesthetics or something or are there somewhat generic ways to just make chat more interesting? Avi Fein: I think there are definitely more generic ways to make chat more interesting. Even now with like hyper parameters [00:23:00] of GPT3 of you can change a temperature and like you don't get back very John McDonnell: like random and interesting are not synonyms Bryan Davis: more or less spicy. Avi Fein: These are like on the spectrum, but like they definitely will have like impact on like the end user experience for people. So you can do these things. I. One thing I wrestle with a lot is this middle ground place where, I talk a lot about individuals and creators and identity, and then you have ChatGPT, which is like the monolith on the other end, and there's an absolute middle ground where you're like what if you take a basket of people. What if you take 10 artists, 10 creators, 10 this, and then you build like an AI agent out of that kind of like they're nearest neighbor to each other. Or maybe they have some kind of value in like this, like mass together. And that's gives you like an interesting output. Yeah, I think it's a really cool exercise to do and think through what are the implications of that and what would be like the end user experience? And I hope that's something we can play around with, but I I don't know. But that's certainly, I [00:24:00] think some things that you could have in like a mid journey type world where it's oh, there's some level of curation that like you do on the back end to say Hey, you know what really makes great art for this area is to do these 10 things. And like we're gonna put those things, those 10 things together and that'll actually move us further down the spectrum from the generic into like where we are on the other end, which is like everyone has their own special kind of, point of view. Bryan Davis: One of the dynamics I think that's interesting is just how bland ChatGPT is, and that's done intentionally. It's by, it's very unopinionated, intentionally politically neutral or driven to be politically neutral. And, any sort of risque content, it shies away from any sort of attempts to basically address preemptively the AI safety concerns. But the result, I think is A consumer experience that lacks luster in a lot of domains that lacks color, that may not be appropriate for a lot of purposes. Avi Fein: It's boring. Bryan Davis: It's boring. So what do you think about the, how do we confront the potential AI risk community when trying to build things that are potentially more exciting, more spicy, more interesting? Avi Fein: You said you take risk you, [00:25:00] you cannot innovate with fear. Like you cannot constantly be worried about the downside risks of something in order to. Unexplored the upside potential of it. Bryan Davis: So let me throw you a curve ball there. Let's say one of your creators is a bomb manufacturer. Yeah. And soMeebody wants to come in, they're like, oh. Like I really want to know, like I, what are the household ingredients I can use to make a bomb? Yeah. This guy's content. Maybe it's I guess you're somewhat limited by the content you're consuming. You're trusting that those sources will be will be doing a lot of the sort of censorship and maintenance of content control. Yeah. But hypothetically, that person could have a private blog where they aggregate these things. Yeah. Just, that just like driving the nail home here. Yeah. What about that case? Avi Fein: I'll actually push back on it in that I think that case is less exemplary because you don't do anything that's gonna break the law or cause like extreme outcomes like that, like where you can have people end up in harm or death or things like that. And those are actually very easy policing types of things where it's yeah, we like, we're not gonna do things that like break the law or [00:26:00] enable to people to easily break the law. There's a lot of other like nuance stuff in between there of misogyny and hate and just like meanness that I think is harder to reconcile with. But I think that case at least is like more straightforward. Bryan Davis: So what do you think the solution is for something like misogyny or hate speech and things like that where there are people certainly on YouTube where they've devoted their platforms to being a******s, right? And I totally understand the sensitivity or the conservativeness of OpenAI and companies like this that are attempting not to build something that kind of perpetuates that. Yeah. What's the, I guess what would be your guide to the appropriate boundary? Avi Fein: I think most of the first order risk is in like the discovery experience or this type of thing where you get into like recommendations of people. And I think for where we are right now, that's less something we have to worry about. Where it's oh, what happens if we like promote like a, a hateful chat bot that like then breeds more hate in the world. That's just not a problem that we're gonna face for the next [00:27:00] like year probably. At least. Can there be those that exist and are created or are powered on it? Yeah. Yeah. And I think that that is a potential downside of opening up to more diversity of perspectives to more, free speech to more of what is humanity and the ultimate criticism of things like ChatGPT, or of Google, or of these like behemoths, is that in the fear of these issues, they water down humanity and they like water down to like very basic, like basic levels and make us all become more like, like, not us all to become more similar, but basically like our experience is to become more similar on those products. And so you end up with they come become further and further from what the world actually is to become like the own bubble where it's oh, this is like the culture of TikTok or this is the culture of [00:28:00] Twitter. And they don't represent maybe the culture of the world as much. And so I think it's just interesting to take a risk and let those things flourish and open up and see where they go and then, deal with the issues that come out of it as they arise. John McDonnell: One thing that I love about the angle that you're coming at Meebo from is your background. Yeah. Cuz you were at Google for a long time and YouTube and then Neeva was founded by people that were sick of Google. And and now you're going in yet another direction. Maybe it's one interesting thing could be the level set on the state of Google. So I, I think we've had this conversation at one point, but what is Google's moat? Avi Fein: They have multiple moats that are incredibly powerful. The first one is distribution in that one of the hardest problems for any company, any startup, especially in the consumer space, is just distribution. How do you, like, easily reach your end customer at scale for like very efficiently and things like that. And the most effective [00:29:00] way oftentimes to do that is to be pre-installed slash the default of something that already has distribution. And Google very early on was the default for Safari. And then, they very smartly took advantage of The wave of mobile and now own 50% market share for Android, where they are the default through, very like many OEM agreements as a the search provider and they very smartly have Chrome. Where they are the default for it and able to, use their own browser as like their distribution. And overcoming that alone is a very haul hard hill to climb and is very independent of the quality of your product. Like you could build something that is 20%, 30%, 50% better than Google. You won't overcome the distribution gap there as a startup. You really need to be 10 x or have some smart unique angle around like why you can penetrate the market in a way that, someone hasn't thought of before. But beyond that, from a product standpoint and technical oh, if you actually wanna go rebuild Google and you think you can solve distribution there's two other moats [00:30:00] that they have. One is, one is slowly getting torn down, which is I'll actually on that one. But the second one is the crawl of the web. They have the entire web content, but not only that, but they have it annotated with structured information about everything there. So they can say oh, this webpage is talking about these people, this location, these topics, imagine if you had that for anything where you could just run a query and say Hey, find me all of the webpages that are about this place, or about this person, or about this idea. That's incredibly, you can't do that today. Yeah, they're. Thousands of millions of businesses spend millions of dollars crawling the web and trying to extract information out of it in order to create a sliver of what they have. Yeah. As just like a baseline default for retrieval and for, other fancier future building. And then the third one, which is eroding now has been their feedback loop in terms of like ranking and that they have query click pairs. So you know that when people do this search that they will click this result in this website and that is the most [00:31:00] relevant, Website to them which can act as a feedback loop into ranking for you and also is very virtuous and just you can effectively leverage that for things like memorization where it's just oh, I've seen this query a million times. So like the next time I see it, not only do I not even need to like do retrieval anymore because like I know exactly what someone's gonna click on, but I can now even do fancier things because I understand a lot more in depth than that. I've known that these words are associated with these like pages and these topics and things, so I can build an AI system on that as well. I think on that one, that's where we're seeing the biggest kind of disruption with embeddings for relevance and ranking, for example. And also with LLMs, which have in many ways like intent, understanding and have content and knowledge baked into them that you don't need to necessarily go rank, the world's web to be able to get the answer out of it. John McDonnell: I was actually curious about your take on embeddings. So one kind of thing that I've had to explain to people about embeddings is that they are, they represent the semantic content of a document. [00:32:00] And something that's magical about Google that's not included in those embeddings is that intent isn't the same as semantic similarity. But it sounds like I guess do you think, are you an optimist about embeddings? Will they end up actually like enabling. That kind of intense discovery. Avi Fein: I'm absolutely an optimist about embedding, and I also think that like embeddings are a reflection of what they've been trained on, like everything else in ai, right? And so if you train a bunch of embeddings on. Google's data, if you had Google's data, they'd be really good for those use cases. Would be really good at intent understanding, like when I say these three words, I know that these three words are probably about, this thing or these, like these sets of things. And that's big biggest issue and that I think people naively see them as This sledgehammer where it's oh, open eyes and beddings. That's amazing. Like I can just use those and you're like no, it really matters what you like, what they've been trained on for your use case. And if they haven't been trained on what you're trying to use them for, they're not gonna be very good for what you're trying to use them for. Yeah. And so I'm definitely bullish of them because I think what is magical [00:33:00] about embeddings and I think what is magical in general about where we are in AI and deep learning. Is you can move into the smearing. And I go a lot and we talk a lot about like nuance and details and like that, like the world is complex. Information is complex, everything is like much more nuanced than we give appreciation for. Words are reductive symbols of that complexity and the minute you can put them into a much more complex like embedding space or latent space, you're able to capture much more complex associations and ideas that are more reflective of reality. And so I think that that and continued investment in that is like a breakthrough thing, but they're not gonna be like the I turn on embeddings and suddenly I have a search engine, type of thing. Bryan Davis: How can small businesses take advantage of embeddings, which seem to be a technology that really is requires first that you have a data set that describes that intent? It describes that purpose. Purpose-built use case? Avi Fein: Yeah. I think anyone who's like starting to mess, that's like starting to play with them. You continue to use like a hybrid system, even be like the [00:34:00] biggest tech companies still use hybrid systems where they're doing keyword based retrieval in addition to embedding based retrieval. Like they're not a mutually exclusive thing. And it's not like one is a drop, like embeddings are a drop in replacement for some of these issues, I would say. And so if you're gonna add 'em in, I would you want to add them in and then be able to at least start retrieving them alongside. And then you can even do evals and say okay is this actually. Doing a good job and augmenting what we're already able to get with keywords. Are we getting what is like in search world, like recall, like we're getting things that we Neevaer knew we were retrieving on before that are related because they're semantically similar to at least assess the quality of your embedding. The second thing I would then say, based on what that assessment looks like for things that are out of the box. And a lot of the things if you're getting into it, is just go do research and choose the right embedding. What's the right model to generate your embeddings from? That's most similar to your use case. You can then try to, potentially take a swack at either fine tuning or even like building your own based on your own like data. But I would say that's like an extra level and really is more dependent upon you getting [00:35:00] extreme value out of that capability. And it's like worth the time and investment to do it because that's not even an easy thing in itself to, to go down that path. John McDonnell: Actually maybe one more question about Google before we get into Neeva, but It seems like a Google should be the big winners here. They have this amazing web crawl. They invented the transformer. And cuz as you were saying oh a model trained on Google's data would have the most amazing embeddings for intent ever. It seems like Google probably should or probably actually does have that. Are they going to be able to exploit those advantages or if not, why? Avi Fein: The problem google and Bing have is when you are the monolith of information. The expectations for users are unbounded and technology's not at that point yet. I don't care how good your data is, I don't care how good your AI is, like we've all seen the fail cases of Bing. We've all seen the fail cases of these, of chat, g p t still and like these other things especially when you're talking [00:36:00] more about facts and knowledge. And when you take that, that that product concept of being like, this is just a universal chat bot that like, has all of the world's knowledge into it that you can trust. And then you put that into a corporation which has a trillion dollars on the line that is not set up for success. Because the amount of risk that you have in revenue that is introduced by this vector, which is Uncontrollable and isn't ready for prime time makes it very hard for you to actually take advantage of the wave. It is said another way, it is the innovators dilemma you like they are at the crossroads of the innovators dilemma, where the risk for them of innovating is so high that it's hard for them to even get started there and the closest they can do is do very quiet behind the scenes things. But at the same time, they're getting just lambasted in the PR sphere for not doing the big things. And you're like the tech isn't there. Like the people who can take risk and the [00:37:00] people who can take the this and produce a good output are the startups, are the people who have nothing on the line. I don't have a trillion dollars to lose. I, like people can go and have a terrible experience with any chatbot and it's really painful for me. But it's not gonna put entire company at stake. And I think Google faces that type of dilemma in practice as one. The second thing I would say, outside of that, they have an absolute leg up from all of technology data, all those other things, side chat experiences are not, chat experiences are not the same. It is not a drop in replacement. People's query behavior. Super ambiguous. It is very generic itself. Google has actually taught you to use few and fewer words over time as you're less expressive in search than you once were a decade ago. And the reason is because they've gotten so good at intent understanding when it's like, Hey, when you put these two words in 70% of the time, you probably mean this. And then 30% of the time, like you'll do a refinement on your search or you'll like, you'll do follow [00:38:00] ups or you'll get to what you want anyway. Yep. Chat from what we said is almost the exact opposite, where people are much more expressive of their needs and much more expressive of their desires and the shape of that data and the shape of the product and everything comes out of it, is actually to be, think quite different than even what Google has. And that is Blue Ocean. No one I think has good quality real chat data for having a conversation like around a topic like you would with a friend. Yeah. Bryan Davis: Do you think that people want to be expressive to entities that don't reward them for that? Avi Fein: You have to be rewarded. Bryan Davis: My, my example comes from my earliest interactions with any sort of chatbots we're often very curious and open-ended. Yeah. And I desire to find the boundaries of their ability to simulate human behavior, simulate chat behavior. And I almost inevitably converge on use cases, which are really mundane. Yeah. Like the way that I use Google Assistant right now is like almost [00:39:00] exclusively to set timers and play Spotify. Yeah, when I first encountered Google Assistant, I approached it with more curiosity and open-endedness. I think I asked it about its emotions, asked it whether like for advice about things. I don't do any of that anymore. Yeah. And I wonder whether we're going through like a bubble of thinking that these technologies are in some way different. But really they're not. Yeah. And like we might all end up disappointed by that. Avi Fein: I think that's a risk. My experience thus far, and this is the optimist startup founder who's headlong into it, is that we have crossed a Rubicon. And it is not that like these problems are solved, but it is that we have the tools in place to be able to create completely new experiences. That break out from what you felt and I felt, and we all felt with Siri, with Google Assistant and stuff like that. We have to deliver on that. Like I, I think that like you have to deliver on that and we talked about this and I like what I think about and what is the most stressful thing [00:40:00] is I, execution is the most important thing. Execution is the most important thing. And how do you simultaneously execute in building out a delightful, magical product? While, you're still flying the plane at the same time and you need people to use it and you need to go to market and you need investors to believe in it, you need users to believe in it. You need to build faith in it along the way. I think that is like a real challenge, but I think the pieces are there to, to overcome it. You can disagree on specifics by evidence. Bryan Davis: Yeah. I don't know if I have a grounded disagreement other than the fact there's like some criticism I think of this potential comes from previous in this conversation. Yeah. Just looking at how things get bland. Yeah. And also I think looking at my own engagement with these things has also followed a wave. It was like initially an expansive, exploratory process and increasingly has become very practical. Yeah. I also wanna counterweight that opinion with the observation that companies like replica were replicant, I can't remember. Replica. Replica, yeah. Which is produced [00:41:00] basically AI romantic partners. Yeah. Went through this massive wave of interest, but I think they maybe have just discontinued a product or something like that. There was some controversial news recently. Yeah. Avi Fein: The, we were talking saying this before, like the most popular use case that I've seen from like data for chatbots is around like adults and intimate interactions and nsfw like the data points that say that is if you look at, searches and queries where, like what are people actually looking proactively for a chatbot? Generally it's that, which is not all that surprising. Like the top search term on YouTube is like porn. The second one is prawn. The same thing for Google, like the, like adult content leads innovation oftentimes. And so replica that was a huge use case for them. And like they turned off like the explicit, I think part, of the capability. And I think the same thing was true for, there was another startup that was like doing something similar where they also had the same disabling capability. I like, there's there's a giant theory all around how this stuff will play out. And I like, [00:42:00] I don't think the Google assistant generic bland, even if you inject personality, but like your assistant now has personality to it. I don't think that's like the starting place for the, for interactions with AI and for interactions with AI agents. It's very hard for me to make that leap beyond the people in tech who are like very interested in playing with these things and like the leading edge, but for the average day person where you say oh, this is actually like a thing that will be used by, people you and I know that are in different fields than tech so to speak. And I think the likely entry point for those experiences, probably more like productivity, utility, more like we were actually getting value out of it. That or the very extreme end where you can get into more of the like, fun slash like emotional empathetic type of things. And that's like talk therapy. Those and some other fun experiences I think are like ends of the spectrum where it'll start and then it'll be who's fastest to converge to complete the rest of it. John McDonnell: I did wanna follow up on what you were saying about [00:43:00] Google's moat and then what Neeva was trying to do. Certainly one challenge that you listed for Google is that, oh, they're just the incumbent and have innovators dilemma, and it seems like Neevaa probably shouldn't have that since they're early enough stage. What's up with Neevaa? Are they gonna, are they gonna win? What's their strategy? How's it going? Avi Fein: I'm no longer there, so I can only say what I as yeah, of course. Know as like an outside of server. No, they absolutely have a ton of opportunity in front of them and I'm really excited for that team. They just launched I'll give them a hype for their own product. They just launched an app called gist. Which has AI basically baked in throughout it. It takes a query and then on the fly assembles an Instagram like story that summarizes the, that topic or whatever you're looking for as an example. And I think one of the real advantages that they have is that they have a corpus of the web. They have that crawl. Yeah. They're incredibly strong team, so they have annotations and references and stuff on it, to know what that crawl is about. They have the power of ai. I think a lot of the challenge for [00:44:00] them will be around the product and positioning and like, how do you find the entry point and how do you find the wedge and what is the experience that wins and what is the audience and product that actually gets you there. And I think that's it. Like from the outset. That's the journey that we were on. And that was the journey I would say, like Neevaa was on. Yeah. Was like, how do you find that? And it was like an incredibly hard thing. Like we all came in, like everyone would just be like, this is a really hard problem. This is a very hard problem. But yeah it was baked in observing these like larger trends that The modality of search in Google was becoming outdated, that the business model was undermining the user experience and that technology was providing tailwinds to try and, provide something new and different. Getting, building up to there is not faster easy. Building out your own crawl thing to go get the internet is not an easy feat. But at this point in time, they have all of those tools at their disposal and I think, it's just blue skies in front of them to take advantage of it. And how's Meebo gonna find a wedge? We're [00:45:00] in the same bucket but less. Yeah. With scrap and hustle and with connecting with our users and building through empathy I would say. No, like in practice, like I said, there's two things I think about every day. Number one is how do we iterate and build a high quality product experience? And I think the path to that is mostly, Getting data as much as possible and through Revs to understand how can we create, the most delightful chats and conversations possible. Yep. And I think number two is how do we bridge the gap while we're building that to get people on and trying it and using it who are partners to us and believe in the vision along the way. Yes. That allow us to build up to there. And those are the two things that I think about most. And then everything else is just an input of like, how do you make that possible. Bryan Davis: Avi, we like to finish our interviews with a recommendation. Yeah. So it could be a movie, a book, a poem, just something that you would like to share that's been like on of top of mind for you recently. Avi Fein: This is [00:46:00] not a great recommendation. I'm sorry, but it's, it is both connected to what we've talked about and what you said before. I find a ton of inspiration from Westworld, from Westworld, season one and season two just only watch those and assume that, the rest is unnecessary . I would say if you haven't seen it and you're interested in this area, I think they did a wonderful job exploring the topics that we've talked about. And the reason that is, is that they very much explored what does AI look like in a diverse world without a monolith. And they do that simultaneous, and they, it gets worse, like as they go on because they have this like Rehoboam, like they have this like central ai, which is like the overall seeing like, singularity. But at the end of the day, what their take is that you have consciousness through individual, unique, diverse representations of ai. Like every it's like a chatbot, but every person is, there's no singularity. There is no oh John and Bryan and Avi we're all using the same [00:47:00] AI model, like behind the scenes. And I think their exploration of that and how the second and order and third order side effects of that in many ways is wonderful. Is wonderful. Yeah. And I think it very much like to me, connects with what I find inspiring about, technology A and where we're at more than anything. John McDonnell: Cool. Bryan Davis: Thanks for being part of pioneer Park. Avi Fein: Yeah, thank you for having me. This was a wonderful experience. Hopefully I didn't talk your ear off too much and it was somewhat interesting . Bryan Davis: Just the right amount. John McDonnell: It was perfect. Thank you. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit pioneerpark.substack.com (https://pioneerpark.substack.com?utm_medium=podcast&utm_campaign=CTA_1) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

10. 

    20 Mar 2023

    Getting kicked out of the SJSU Food Court with Peter and Chris

    Peter Lowe and Chris Hockenbrocht discuss their startup Fresh Bot, a food automation platform that uses robotics and machine learning to reduce labor costs and make food more affordable. They discuss the importance of "jedi mind tricks" when launching a business, the trend of unhealthy food in America, the potential of automation in the food service industry, the challenges of automation, the difficulty of hardware startups in Silicon Valley, the potential of automated delivery, the idea of a burrito cannon, the technical risks of building a restaurant automation platform, the importance of owning the experience, their own diets, the idea of eating what our ancestors ate, the Amish and their cautious approach to new technology, the limitations of reductionism when it comes to food and nutrition, and their shared values and goals. - Chris and Peter === [00:00:00] hi, I'm Bryan and I'm John. And we are hosting the Pioneer Park Podcast where we bring you in-depth conversations with some of the most innovative and forward-thinking creators, technologists, and intellectuals. We're here to share our passion for exploring the cutting edge of creativity and technology. And we're excited to bring you along on the journey. Tune in for thought-provoking conversations with some of the brightest minds of Silicon Valley and beyond. John: Welcome to Pioneer Park. Today we're shooting live from South Park Commons. Our guest today are Peter Lowe and Chris Hockenbrocht. And Peter is an expert in hardware and product. Chris is an expert in machine learning and cryptography. They're both members here at South, Park Commons, and they're building a new startup called Fresh Bot. Peter and Chris, welcome to the show. Chris: Hey, thanks John. Peter: Welcome. Thank you. Glad to have you. How are y'all doing today? Bryan: Good with a little bit of setup for our first live feed. You know, both of you were here for some of that, so we're working out the kinks of getting [00:01:00] on microphones and getting videos set up. So, uh, you know, first time's a charm or maybe the third time's a charm. We'll find out. Chris: Yeah. John: Yep. All right. So we're super excited about the work you guys are doing and it entails both robotics and food. So, do you wanna tell us a little bit about what you're working on? Chris: Yeah one of the things that we really see as a trend is food costs rising. And so one of the questions is how can you even reduce that? And the way we see tackling that is through automated, front end in food service. So I wouldn't call it robotics, but a lot of different automation techniques that can be applied to different sorts of food preparations that hopefully can reduce the cost of labor going into the food. And hey if we can solve that, then we can start to think about bringing down food prices. Bryan: Interesting. Yeah, I just read recently that there's a suspicion that there's some collusion in the egg industry that is causing the massive rise of if egg prices that we've experienced the past couple years, [00:02:00] but obviously that's further up the production pipeline than what y'all are doing. So concretely, what is Fresh Bot? Chris: Right now? Currently we're looking into a variety of different products for food automation. We have a MVP on smoothie automation and other drinks. So there's a lot of different PE components that we put into a machine and it allows us to dispense liquid solids, do blending. And so we could conceivably put a lot of different things. One of the things I really like about this is that it's customizable. So you take individual machine and we can stock it with different things and we can tailor actually to the particular market. But we can do liquid solids powder dispensing. We can recombine these into any sort of drink that you might want, Peter: One reason yeah, starting with this kind of drinks platform and starting with smoothies, which are one of the hardest drinks to make is interesting. Like for reference Starbucks and Dutch Bros, like about 75% of their drink sales are their cold beverages at this point. They're, cold brew coffee, [00:03:00] frappuccino you know, juice drinks and stuff. So, I all of that is gonna be very easy to automate with the platform that we're making. Just for a little bit of market orientation reference there. Gotcha. Bryan: And I recall, so I think several of us have had the pleasure of being part of some test exercises with Fresh Bot, and it wasn't exactly Fresh Bot, but it was Peter testing your smoothie recipes here at South Park Commons. And at the time, I believe you just sort of brought in raw ingredients and you were just mixing on the spot and sort of having a few different offerings. And I guess that was just sort of a, a menu taste, a menu testing. Is that right? Peter: Yeah, yeah. I mean I think this kind of comes from having gone to the Stanford D school and taking on this product mindset which is, has been a sort of useful mindset and tool set, it's very difficult. Hardware is so complicated and so difficult to make that your engineering instinct is that you want to start building something immediately. But that's not necessarily the fastest way to get the answer to the questions that you have, right. About a startup addressing whatever your key risks are. And with a lot of the prototyping that we've done, it's actually. Not necessarily involved a soldering [00:04:00] iron at the first blush. Right. You know, one of the key questions was like, are people interested in food and the venues that we're interested in, do they want food or what, maybe which items resonate more with people, you know, did they want the sugary thing or the healthier thing? You know, getting some of this broad thick data, from users about like how they think about food, what they like. Bryan: I love you've shared with me over the past month or so, some of the stories from the front lines of your testing. I think some of them are really fascinating. How many places have y'all been kicked out of so far? Chris: Well, I mean, as far as I recall there's been two. We went to a mall, it was a security guard. He came up and said, you just can't be doing this here. Right. Bryan: I guess we should, uh, we should give people the setup. Mm-hmm. So what are you doing when you go to test these on site? Chris: So yeah the machine was taken to a mall. It wasn't actually a fully functioning prototype. What we were trying to do is gauge interaction. Would people simply walk up to the machine, interact, attempt and order Peter: mm-hmm. . And this was sort of not a machine, it was really sort of a fridge with a sticker on it. Yeah. It looked like pre-engineering. Yes.[00:05:00] Chris: Yeah. And security wasn't very happy about that. But you know, the only regret I think we have is not walking out in handcuffs, hey, it would've made for a great great PR stunt there. The other time was we were more recently at San Jose State University. We went right into their food court and we successfully got about two hours of sales done. Students were coming up, people were enjoying it, and then over time, one person would come up, they would go talk to their manager, go talk to this person. And eventually the building manager came who was in charge of all the food court. And he said, you just can't be here doing this. Like, you know, essentially people pay to come in. Like the restaurants that are there paying, you can't just come in. And Peter here was doing a really great job of deflecting them. You know, just, just, uh, it's really great if you change somebody's focus they start thinking about things in a whole different light. Like, if they're like, what are you doing here? Well, we're making healthy smoothies for people and then, you know, we really [00:06:00] care about people's health. You know, you end up in this place where, now they're like, pitting two goods against each other. Either I'm doing my job or I'm like supporting Healthy Smoothies. It's this cognitive dissonance that they have to resolve. And so it wasn't until we got to like a really serious manager who just came and told us that we had to leave that, uh mm-hmm. Peter: Yeah. Just to be clear too, I I have the, food safety handler Safe Serve certification. We're not breaking any food safety rules with any of this stuff. We do take health and you know, proper process seriously. Yeah. So yeah, we just can't pay the rent. Right. Yeah. Early testing phase. John: Jedi mind tricks are crucial to, to launching this kind of business. Peter: Yeah. I mean, I suppose really any startup, I guess there's a good reason why, you know, YC asks essentially, what's the biggest sort of non code, hack you've ever pulled off. Right? So there's a lot of hacks necessary sometimes. Yeah. Yeah. Absolutely. Bryan: yeah. So I'm curious to connect this back to the larger theme of health and access to [00:07:00] healthy food in America, and whether or not your efforts in this area are based in some sort of critique or analysis of what's happening in that space. Chris: Well, there's certainly a long running trend of food towards less healthy things, and there's probably a few different components playing into this. One is just taste preference, right? Less healthy food tastes better. People like sugar. Sugar. When it sits on the tongue, it is just, hmm, that's good. And it's hard to avoid. And so the products that you end up seeing at the supermarket, it CPG that is, or the products that you're getting from any sort of restaurant might be laced with additional sugars or additional fats. Things that just really, make it taste good. And so it's hard to satisfy the desire for healthy and balance that with taste. Another factor is, there's this industrial farming situation where we have a bunch of subsidies that go towards different sorts of crops and being subsidized now and being produced in mass. [00:08:00] Well, why don't we just shove it into everything? Soy is a really key component. Corn is a really key component, and corn doesn't just satisfy food things with high fructose corn syrup. But you, there's a lot of non-food uses of corn that are really subsidized. Ethanol production is a big thing that goes into gasoline, and so there's, I would say there's a little bit of a incentives, misalignment for actual food to be healthy. It's, and the question I think still remains like, will people really accept healthy food on mass? Which. We're trying to get to, one of the things that we're working on is perfecting recipes that actually taste good, which still have a good balance of health. Just not putting additional sugars into things. You know, no high fructose corn syrup. No other weird additives, preservatives. We want to focus on just really presenting the base foods in the healthiest manner that we can. Peter: Yeah. One [00:09:00] a thing that's interesting to think about is, you know, Chris mentioning that our palates are naturally warped to want, salt, fat and sugar ratios that, are not healthy for us, and there's the perverse incentive because those are very cheap ingredients, for third parties that are preparing our food to just load up on that. Cuz that's a cheap way to, to bulk up the product and it tastes good. But we also know that it's not healthy, which is why we have this, kind of mental mindset that we go home to eat something healthy, right? You make something that has the amounts of that that, you know, that would be good for you. Maybe it doesn't taste, quite as rich but that is, what we know that we're supposed to do generally, at least, depending on what group you're part of. It's a gradually growing awareness of that. I believe it was the the National Institute of Health recently published a report, but basically we need to eat, more fruits and vegetables is essentially their largest prescription to move the standard American diet toward healthier direction, reduce the amount of diabetes and obesity and heart disease. But one thing that's interesting is, that we've thought about relative to this a bit is working with like lower quality ingredients, is a way that you can save cost. And, having less operations happen in a restaurant particularly something like a large fast [00:10:00] food chain, which is kind of relying on the very lowest paid, tier of labor right? Is also a you save cost, but with kind of doing less of the preparation fresh, because you honestly can't run a chain that has tens of thousands of locations with people who aren't paid enough to care. Right? You need to take the control away from them, move things upstream. Things are now less fresh. Maybe you're working with lower quality ingredients to save costs too. And so the food doesn't taste good naturally, right? And so it needs to have all kinds of alchemy applied to it. Particularly adding a lot of salt, fat and sugar to make it taste palatable. Which is kind of how we got our our sort of fast food situation that we have now. But one thing that is sort of interesting about how maybe technology could be applied here to fix this is if you can engineer preparation techniques using automation, that will prepare the food the right way, every time you can trust them. I mean, they will work as well as they're engineered too. You can then push more of the food preparation out to, the point of consumption. Things are now being prepared more fresh. They taste better because of the very significant cost savings this can offer on labor and other parts of the overhead costs, [00:11:00] right? You can still be extremely profitable, but use a higher grade of of ingredients, so that it tastes better. And so that can be, a fairly large lever just to essentially make good food taste better. I was recently running a test in central Pennsylvania just at a, essentially a cafe and making different smoothie recipes and things for the owner and trying them out on customers. And, know, the owner just couldn't believe that these things were not made with added sugar. And so he'd asked for the recipes cuz he, essentially didn't believe it could be done. That you could make things that taste this good, that don't have the fillers and junk that a number of things are reliant upon nowadays. Yeah. John: Yeah. Just playing back. So what you're saying is if you use low quality ingredients and you have low skill staff, you essentially make up for those things by filling your food with a bunch of salt, sugar and oil. And it doesn't have to be that way. Yes. Okay. Chris: Yes. Bryan: So I guess one of the things that's relatively alive for me right now is something from the All-In podcast and some of the more libertarian leaning members of the [00:12:00] Silicon Valley tech scene who have criticized the minimum wage and the growth of minimum wage in California specifically for restaurant workers. I believe that there was a bill that was proposed, To allow specifically fast food workers to have a higher minimum wage than other industries. And you know, this is a lot of back and forth and maybe there's goodwill there to like, you know, looking out for the, some sector of employees, but then there's also this fact that why only this specific industry does it make any sense at all? And some of the people that I've heard talking about this, were criticizing this, saying that this is pushing the incentives of automation or the incentive of restaurant companies in the direction of automation faster than it would otherwise be. And I'm kind of just curious to hear, what are the trends in automation of food processing from farm all the way to table? What are we observing nowadays? Chris: Well, it's really complicated and you're touching on a key point, but it's already happening. We have to be real, like, They are pressing for a increase for large [00:13:00] chains fast food chains from the $15 minimum wage to 22. Now the way I feel about that is a little bit mixed. I mean, obviously to your point, we want to see people be paid fairly, but at the same time, food costs too much and we're just making it cost more. And the people who eat fast food predominantly are people already unable to pay for good food. So honestly, if they're getting the bottom of the barrel and now they're having to pay more, we're pitting two goods against each other. And to the point of, automation, well, McDonald's already doing it. At the beginning of this month they launched their semi-automated location in Texas. No, it's not fully automated, but you don't see people, you order at a kiosk front end. and you get an automated delivery. The thing about burgers though, is that they're really hard to automate. You know, you end up building a Rube Goldberg machine, and one of the concerns they have about that specific [00:14:00] food class is, how reliably can you automate it? But the truth is, it's happening. They're probably gonna push forward on it. They probably have the capital to invest in specifically burgers. And I see this going for every food chain. If they can't build it, they'll probably buy it because there's already other people out there making these things. There's creator, there's miso, and they're specifically focused on these chains. Now, one of the factors that has to be accounted for in the supply chain for fast food in particular, is that they already centralize a lot of their labor. A lot of the patties are preformed in central depots, fries are precut. In-and-Out is a little bit of an outlier in that they, you can see them take the potatoes and rip the thing down and get the fries ready. But most of them, all the ingredients are pre-prepped, just ready to throw in a fryer or on the grill. And so one of the questions is like how do you solve that last mile issue in terms of automation? How do you make a burger flipper? How do you make a fry cooker that's [00:15:00] fully automated? And how do you then get those packaged up into your little paper basket and served in a bag and the way that people are at least expecting today? And that's the, that's one of the things that I would be, curious to see where it goes. Cuz that's not a food class I think I want to personally work with. It just seems very Rube Goldbergian to be back to that. John: Well, maybe an implication of that is, is it possible that, with these movements, say the minimum wage gets much higher, McDonald's traditional method doesn't work as well. They try to switch to these Rube Goldberg machines. But it's possible that hamburgers are just not a good format for automation. Is it possible that they will essentially be counter positioned by other automated food options that are just much more amenable to being automated? Chris: That's our thesis. Yeah. You know, we are pre-selecting foods that we think can be fully automatable, and I think one of the risks that we have to work on is understanding people's food preferences for these [00:16:00] categories. But yeah, but things like soups are actually a lot more highly automatable. We've been working on drinks. Drinks are something that's actually relatively straightforward. Other groups have worked on bowls, keynote bowls, the kae bowls. Yeah. Those seem to be more amenable to automation than any of the, the current fast food offerings. I think, fast food had to do with a cultural zeitgeist that existed and is only continued like the burger is American. Or you know, with KFC and the Colonel's fried chicken, he made a method for fried chicken in seven minutes and that was his thing. And it played to the tastes of people. Now we're in a period where tastes are changing, where we are rapidly encountering all sorts of foods. I think one of the things growing up in a small town that when I go back and go visit very small towns along the way, driving through I've noticed is that there's now a lot more global food options. And it's not just the cities that have these options. It's everywhere. And so I think that there's a [00:17:00] first, a preference and change that allows us to even consider these things. But from an automation perspective, , there's a class of foods that are going to be automatable and there's a class of foods that really are, you're at very least gonna struggle very hard to make them work. Mm-hmm. Peter: Yeah. There's kinda just a broad idea in robotics or I guess automation in general of like structured versus unstructured environment. Right. I mean, why data is so automatable is because it's very structured, right? As, as rules, you can interact with it. Mm-hmm. You know, the physical world tends to be, unstructured, which is, why we have yet to see useful home robots or other things. Right. You know, there's unpredictable things. We still don't have autonomous vehicles, we don't know when that'll happen. And so, you know, if you're dealing with, particularly unstructured food or the sort of, the more unstructured the food is, obviously the harder it is to automate. If you can, deal with food that tends to be more structured. That's the lower hanging. Fruit, no pun intended in terms of yeah. You know, you're gonna be able to reliably automate at scale. Bryan: Mm-hmm. , I'm curious, obviously this is a little bit out of the purview of the work you're doing now, but [00:18:00] one thing that is sort of, a holy grail of automation and for of the food chain for me is thinking about the massive numbers of people that are employed, just picking vegetables and doing, probably the lowest tier work in America. This is work that is overwhelmingly immigrant labor is overwhelmingly used for this. It's, you know, we're in California just a, a hundred miles into the Central Valley, and we can just be in the densest agricultural sector maybe in the world. Do y'all have any sort of insight opinions about the way that automation, or the degree to which automation can affect the low skill labor of picking, picking strawberries? Mm-hmm. , which is to me some like holy grail of. Sensitivity in robotics and making robots really work at that level? Chris: Well, I just saw the coolest thing on YouTube the other day. It was a group of researchers working on gecko skin which combined with some particular joint mechanisms will allow effectively plucking a fruit. Now, like to your point, there's always [00:19:00] been a sensitivity issue, but I think with the combination of the gecko skin and this particular sort of joint that they're using, it's very conceivable that they can softly pull things. But the greater trend that I see is actually more in robotic robotic driven and vertical farming. So I think the thing is right now you have an existing model of a farm and do you want to apply robotics to it? You can turn the problem on its head. And this is what I think where a lot of the success will come from is actually designing the farm for the robot. . And so you certainly have to solve these problems of like, how do we pluck fruit in a very careful way, but you're looking at large tracks of land that are spread out, like the efficiency really comes, I think, from compactness as well. And so vertical farming is, I think one of the other key pieces to this. It's really unfortunate. I'm so excited about this. I've been excited for about a decade about this and just [00:20:00] wondering, you know, how to push forward on these things. I think there's a lot of groups that have this solved though, so not a problem I'm eager to touch right now. Peter: Yeah. Which relates to that idea of, you're structuring the environment, you're structuring the farm to make it more friendly for an automated system driven by logic and whatnot. The desk in my office I acquired from the liquidation sale of Abundant Robotics, which was a strawberry picking robotics startup. Last year or so they went outta business. And yeah, I mean it's, some of these things can be very hard to do. Seen some things recently about a soft robotics company that's essentially using the compliance of their gripper, which is, made out of alasteric material to just help account for the fact that even pose estimation and grabbing stuff is a pretty hard problem. And that's actually some stuff that I worked on a fair bit in the robotics lab for some very large tech company concerned with this sort of thing. Essentially, grippers that essentially have mechanical logic that makes the actual grabbing easier because the gripper is so grippy. Bryan: I think it's a good segue into talking about hardware as a [00:21:00] startup, like, I think in Silicon Valley there's a really strong software bias. There's really strong reasons why software is preferred. It's way easier to test, it's way easier to build. Hardware is hard problem. Chris: Margins are also higher, you know. Yeah, yeah, yeah. Well, what would you like to know? Bryan: I'm curious, what gives you a conviction to pursue something that, you know, is sort of hard? Is it this sort of thesis about the opportunity? Is it a thesis about the impact? Is it maybe a fascination with the technology? What kind of drew you into the problem made you willing to commit to that harder problem? Chris: Well, for me personally, I can just sum this up real quickly. I'll say it's incremental, but monumental increment that automation can offer food service. We've been talking between ourselves about automated delivery as a particular thing that's going on. And we've been watching for five or six years is all these companies pile money into just automated driving, self-driving cars. [00:22:00] But when you talk about delivery and there's a number of different mechanisms of delivery, like we can talk about trucking, long, long haul trucking, or we can talk about taxi services or we can talk about food delivery. I think each of these has some really big problems, insurmountable problems that really rely on humans. And well the question always is how do we get the food to the human? But for us it's about how can we place it in a space that can prepare it for you. How can we be small, nimble and put a machine down that's really highly functional, whereas we're not gonna solve, like driving a car and then, having to deal with the actual drop off problem, right? Mm-hmm. , like you think about being in a city with automated delivery, if you're on VanNess and your customers there, like where do you park? And then who takes the food from them? Do you park two blocks away and force 'em to come? Like, that's even worse than the current solution with a human. So, how do you get the food from the car to the person? And there's just a lot of [00:23:00] things where there's a human need. Peter: Yeah. From talking to some actually other SPC members who have worked on this problem and you reading about it, we have particularly applied to this question of food where like some discussions we've had with investors, where they've been excited about our approach because, the default assumption about the future of food, at least a large part of that future, was assumed to be, we have our automated delivery overlords. Right. And this stuff, comes by, whatever automated, planes, trains, and automobiles, you know, the burrito flies in your window from the regular restaurant that's, across town and taco Yeah. Taco copter... Bryan: right, burrito cannon, you know. Yeah. Peter: But it was sort of like, you know, right, we're gonna have this, it's gonna be so cheap, it's gonna be ubiquitous. Like this is the default future that we sort of believed in and accepted. And then, essentially the magic fairy dust to, to make this technology come, has been very slow incoming. I mean in the sense of solving, know, the edge cases for automated driving. But then there's also just like practical constraints. Like, you know, there's still two to $300,000 of special hardware to make a, an [00:24:00] autonomous cargo, right? And that's, it's not necessarily clear that that's just gonna become, much cheaper immediately, which makes the viability, the economics simply very poor. You, or maybe you're cheap drone, know, it's like, well, okay, you know, there's, there's regulation. I mean, what, what about when it falls on people's heads and kills them? Or, you know, the constant buzzing, right? Like that seems a bit stuck. So kind of saying, okay, so maybe this sort of thesis about the way the world was gonna be it appears as now kind of coming crashing down as, Argo gets out of this, Amazon gives up on the delivery, right? Like, we're all kind of like, returning to earth on, what's really possible there. And besides just the problem too that like, who loves when their food gets delivered and how it's not the right temperature, right? I mean, maybe you can help with that problem, maybe you can't, but it's not fresh anymore. That's kind of the bottom line. So this alternative thesis of, okay, so maybe the food is actually made on site, right? And you use, technology that we have a pathway to build it today and maybe that's more what the future is like. It's local and automated. Chris: You know, one of the things I love that you touched on is this capital, right? Like from [00:25:00] a, if a delivery car or automated delivery car costs a quarter million, like that's a huge capital investment for something that's gonna be gonna achieve 10 20 orders a day. Cause it still has to go to the restaurant and pick up the food and take it to you. That's a 30 minute, at least round trip. What we can do is if we have a machine that's on a vending machine, capital expenditure level, like maybe it's a little bit more because there's definitely a lot more going on in it, but 10 to $20,000 for the same price, we can put down 10 15 of these machines. And well now they're just everywhere, and you don't think about where am I gonna go get food? There's a machine that offers something that's close to you. Maybe it's in your building, if you look in an highrise or maybe it's, in your office space or in the hospital, or X, y and Z. There's, put 'em everywhere. Peter: Yeah. Yeah. An initial problem we're thinking a lot about, right? Like right here in this building. If we want to, if we wanna get food, which we, get lunch every day, we have the unique advantage often of having food, shipped from across town, from a [00:26:00] good restaurant. But if you're in the typical office worker scenario, I need some lunch or a snack, right? It's like, what's the nearest thing? Well, a couple blocks that way, there's massage. That's a pretty good, reasonably healthy, affordable ish lunch option. There is like one burrito spot, but then other than that, it's like grilled cheese. There's like the fancy French cafe, if you want a smoothie, there's really no place to get a smoothie. But essentially your lunch options are pretty poor here. Like, if you work here you're gonna be pretty dissatisfied pretty quick. And that's true a lot of places there's just so many places that you spend time and regular restaurants aren't serving them because maybe it's hard to find a location. I mean, you know, building a restaurant, for a typical fast casual, you're looking at on the order of half a million bucks you can't really operate a restaurant with less than two people. So that's at least $500 of expenses a day. Versus the automated economics. It's a totally different picture. And so it, it completely changes the opportunity landscape. John: So you've kind of talked about the technical risk that you're not taking, which is, you're not automating delivery. What do you view as being the technical risk of this endeavor? Chris: Still, they're machines and they have to be reliable. [00:27:00] Like, think 9, 9, 9, 5% reliable, we want a failure out in one out of 2000 to 20,000 cases because you don't want to have a problem where you have to keep going on site just to fix things. Now if you have to restock, that's a great opportunity to, to refix. So you want, you probably want your failure rate to be a lot lower. And so it comes down to testing, and testing is time. And that's, I think, really the difficult thing about bringing a product like this to market is it has to be pretty damn reliable. John: . And is it the case that you're able to assemble what you want out of off the shelf components? Peter: Yeah. There, there are some key systems that we're working on, getting IP for for like this initial platform. Each platform is going to rely on at least some of that. And especially as we move into, our later food categories, it's gonna become more and more technically complex. As we move in, into other types of food that may be less straightforward to automate. John: So if the parts are available off the shelf, how come no one's done this before? Chris: I mean, you still have to [00:28:00] solve for reliability. There's another factor that there's been a regulatory inhibition. This hasn't been a poet class that the regulatory frameworks have really addressed. The recent developments and the regulatory frameworks have only just allowed this to come on the scene. And it's still not even universally true across the United States. There's still food codes that need to change to accommodate the existence of this product category. Peter: Yes. Yeah. Another thing too, just in the sense that we're dealing with automation, robotics and food, there's a lot of technical discipline that's necessary to execute this, but then there's a lot of product type thinking, culinary type thinking, et cetera. It's much more difficult than building your typical SaaS product in terms of the amount of expertise that you need to consult and things that you need to not f up. And so from the earlier waves of companies that have tried to enter this space we've seen a lot of key errors. Probably the most common one has been the food hasn't been very good, right? They've been working so hard on getting the machines working that something key gets screwed up [00:29:00] with the culinary aspect of this, right? And from seeing how these companies have talked about themselves and what they focused on, right? They were, they thought of themselves as robotics companies. And that's not what you are, if you're serving food, like the customer ultimately does not care right outside of how it affects their experience. The thing they're getting from you is food. The food has to be good. Health matters, cost matters, convenience matters, right? How you get there is not ultimately important to them, right? Bryan: So I'm curious whether you think of yourselves as building a platform for people to enable their recipes to be served, or really whether you want to take the responsibility of develop recipe development as well as the product development, as the robotics development. Chris: I think it's both, right? Like flexibility of having automated food. is that we can offer the chance to customize. We're talking about healthy food, right? But what if someone does want to go, have fun one day let's make it a little bit more sugary today. Or, you know, up the salt, why should we stop them? You know, people are ultimately [00:30:00] responsible for the health. And so by having this system of automation, there's a lot of key points at which people can start to customize to their own flavor profile. In fact, we think that personalization is one of the key things down the road that is something that we can facilitate and will make this more attractive. Peter: Yeah. But with respect to that question, we believe owning the experience is gonna be the way to really unlock this opportunity, particularly at first until we have a very good framework. So ChowBotics was one of the early companies in this space. And they, rightfully were hesitant about owning both essentially the culinary aspect of their machine as well as the technical. And so they wanted to just be an equipment provider. Their machine was essentially an automated salad bar, was what the intention of it was. Sold the machines for about $30,000 to a customer who was then responsible for running the machine, restocking it. They had a relationship with a big food supplier, so they could, order things to fill the machine with, but they were responsible for all aspects of that experience. And from the conversations I had with customers, that was not a good experience [00:31:00] for the venue customer which then trickled down to the eater customer. It was not a good experience for them. And so ultimately ChowBotics was acquired by DoorDash possibly not with essentially a giant exit, which was then subsequently shuttered about a year or so after acquisition, right? So it's essentially the experience was not strong enough to get the kind of growth to really justify um, the investment that had gone into the company. So, anyway, the experience is really important and we are intending to own that at this point. Bryan: Hence you perfecting your smoothie recipes on a live audience, Peter: so, yeah. Yes. Bryan: Yeah. I'm curious to kind of dig in a little bit to your own diets. Do you make a habit of trying the latest in food technology? Be it hue or, you know, uh, yes. Or well, or either of you been vegetarian or vegan or, uh, carnivore? Chris: I have a weird thing, you know. First off, I have to say, I don't have a problem eating the same food every single day [00:32:00] in a row. When Soylent first hit the scene, I was incredibly excited because I'm like, this solves a problem for me. I don't want to think about having to go get food and I don't want to pay for a meal. Like, this is cheap, fast. I don't have to think about it. And it's ostensibly healthy. Ostensibly healthy. Let's be clear there. I started doing this before it was actually a mass market product and there was this whole community around DIY Soylent. And so I did the people chow, I don't know if you guys know about this, but there was a whole series of recipes that people put together. So I and a couple other people I knew in university did people chow. We would mix it all ourselves and like have a weak supply or whatever. And when Soylent that came out, when I finally started getting the subscription, I was like, you know, this is good. now, there was a lot of variation as they came up with different versions. Some of it wasn't particularly upsetting to my system, but some of the later versions, especially as time went on and I was running a startup, like it was kind of a little bit violent on my stomach. It caused a lot of inflammation. [00:33:00] It was very disruptive to, I think, overall gut health. And so after about a year so I'd probably been having swelling two years maybe, but and how much of your diet? Not completely. So it was about one to two meals a day. But at some point the recipe was just so bad. I just couldn't do it. It was just, it, I like version 1.4 I think was like the magic recipe in some sense, but they started really going really soy heavy and there was just a lot of disruptive factors in it. It, I don't know that a lot of people can do it. Might be a good and a pinch now that you can go to seven 11 and get one, but, it's not something I would really choose to eat. Peter: Yeah. Yeah. I did a, I said I did a diet in college. I was diagnosed with Crohn's disease didn't want to go on the standard treatment cuz my brother had had a sort of a mysterious health problem, been exposed to that and then developed the known type of cancer that's linked, was linked to that type of drug. And so when I was actually diagnosed with Crohn's and the doctor was saying, okay, so this is the, treatment. I was like, [00:34:00] that's, um, not appealing to me right now. Given that this was, whatever diagnosis had been whatever about two years ago. My brother is fine now, thankfully after, a lot of treatment and and whatnot. So I was interested to find out about any other alternatives. Found out about a diet called the specific carbohydrate diet that's actually been around for around a hundred years. It was kind of developed empirically essentially just, trial and error for primarily children with Crohn's and even since I had done the diet microbiome science has become more developed and it's, become more understood about how this works. That essentially by restricting certain kinds of carbohydrates that are favoring the particular loss of biodiversity and an upset that's happened in the biome of someone with one of these autoimmune digestive type disorders, right? You're essentially selectively re-reading the microbiome by disfavor the bacteria that are causing the inflammation. So it was very effective for me anyway, but it was very, it required essentially a fanatical strictness of adherence, because, you give them one shot of their fuel and they're powered back up again.[00:35:00] So essentially like weed these guys down to almost extinction and let the natural order recover. It took, about two years during which I was having to cook a lot of my own food. Because one of the things I had to avoid was added sugar, right? It's like that's in, you read labels, that's in almost everything. Also grains and starches. So, and it was basically super effective for me. Got me really good at cooking. Yeah, so for me that was the big eye-opener of like, wow diet can really shape health outcomes. Particularly if you have a disorder. But anyway, not to go too deep into that, but that was a big inspiration for me with diet. Now, I mean, I'm open to trying new kinds of things. I'm gonna have some impossible patties in my freezer. I'm actually not eating them right now. I'd had a couple of 'em and then kind of got a weird headache and I don't get headaches very often. Okay. Suspicious. I don't know if it was, related to that, but anyway, But but despite, essentially our focus on robotics and automation and all that I'm actually kind of a Luddite when it comes to my beliefs about food in the sense that, this food that we've been eating, since, since we began, since, humans were [00:36:00] there's a lot of optimization that's happened. Right. And, given the complexity of biological systems, I don't know if we'll ever really understand, everything that's important about the way that these natural foods interact with our bodies, right? Mm-hmm. , you know, there's probably, things and things and things beyond microbiome and all that about the way that food nourishes us, that it's ultimately like, This food has nourished us and our ancestors very successfully for a very long time. Also, we're used to eating it and we like it. So there's not so much the the risk of the seems to be reckoning right now with know, kinda the fake meat where it's like, okay, maybe, consumers are not really adopting this. There's this question of is it as good for you? Is it just too strange to be eating fake meat? Mm-hmm. , not that I'm inherently opposed to it, but it is a risk of adoption. Yeah. John: So when you say you're a Luddite, does that reflect a certain perspective on what a diet should be? Like? Does that look like Paleo or is there a way that you would summarize it? Peter: Yeah, yeah. I mean, I guess just in the sense like, you know, and Ludi applies too to standard American diet. I mean, in the sense that you like, look at, look at any black and white picture from your [00:37:00] family's history. Right? Those people are all really skinny and like healthy because this diet that's arisen over the past 50 or 70 years, right? That we're so surrounded by that, we think it's normal. It's not at all normal. It has nothing to do with our history. And we're eating this way and we're getting diabetes and obese and cancer and cardiac problems, right? I mean this diet has nothing to do with humans and what humans eat. So I guess a Luddite in that way essentially that kind of the simple, like if your grandparents don't recognize that as food, like Right. It probably isn't. Yeah. Bryan: I think Michael Pollan's takeaway as something like, eat what your parents eat mostly vegetables. Yeah. Maybe it should be extended to eat what your grandparents ate. John: Now the parents aren't a good Exactly. Anymore. Bryan: You've had a generation of people raised on factory farm food or whatever, processed food. . Chris: Like almost two or two generations now. Two generations. Yeah. Mm-hmm. sixties, seventies is like when this really came about. Peter: Yeah. Yeah. I think he says "eat food" and then he, gives his definition of food and it's like, mostly vegetables not too much or whatever. This is sort of summary of his philosophy, which Yeah. I mean, it's [00:38:00] sort of like, if it ain't broke, don't fix it. And it's, it's clear that what we're doing now is very broke and what, know, our ancestors did is not very broke, you know, blue zones and all that stuff. John: Yeah. It is kind of mysterious though, cause it's not exactly the case that our ancestors ate, just like what we would now consider to be health foods all the time or something. Yes. I mean, like, if you look at it like, at the traditional breakfast is like greasy eggs and bacon and potatoes or something, right? Mm-hmm. gross. But there's like something about, there's something about the way people were eating or I feel like there's something we really don't understand about why that was like reasonable and like the new Peter: stuff. Yeah, we had the cholesterol craze, you know, at some point where it was like, oh my God, eggs are bad. Chris: One of the funny things is a lot of this has to do with some trends in the temperance movement 120 years ago. So, wow. Breakfast wasn't a category that we had, it was very indistinct from most other meals until we started formulating, cereals. So there was this man named Kellogg, of Kellogg's fame. . Okay. And [00:39:00] he he had this sanitarium, he had some really weird health ideas. And one of his things was like I'm trying to figure out how to say this politely. Eating cornflakes stops men from masturbating. Okay. I'll just put it like that. You know, like that was part of his philosophy. And, It probably wasn't a good marketing tactic. Maybe it was in the highly Christianized, pious world of there. John: It was Victorian. Mm-hmm. . Chris: Yeah. Yeah. The Victorian sensibilities. Peter: So the Amish my parents live in Pennsylvania not far from like Lancaster and places where there's a lot of Amish. I'd read something a while back about, the Amish, which at least according to the group that this journalist was studying, it was kinda like, they're not inherently opposed to new technology, but they're very cautious about it. And so if there's like a, an enthusiastic early adopter person, they get to try the thing and then the elders watch them and then try to come to this holistic conclusion about whether that thing is truly good for their life. And, it turns out, I guess they usually decide not, but you know, that the Amish are much less expo. I mean, the sense of like, you know, how many Amish, kids being raised on iPads, who don't have social or you know, hand-eye. I [00:40:00] mean, I wonder what's gonna happen, over time as we get, deeper and deeper there could be divergence. Right. Really . Bryan: What would be y'all's rule of thumb for the way that experimenting with diet makes sense. So for instance, like the critiques of fake meat, the critiques of huel and of soylent, it these sort of things are, or almost like scientifically produced and derived versus something that maybe the emergence of a fusion cuisine, like the sushi rito. Right. Do you personally have a rule of thumb that you're like, okay, like that still fits some model of like food versus this is like scientifically produced protein matter? Chris: Well, I mean, we have to just question just science know. Anything you know about food is just complicated. And one of the things that, I think scientific fact that really speaks to our ignorance is we, our bodies produce about 30,000 of the 300,000 necessary proteins for us to survive. Where did the other 270,000 come from? And, one of the suspects now is that [00:41:00] it's really the complex gut biome mm-hmm. , that they're responsible for doing, for producing the requisite proteins. And just getting into the exploration of that, we're just touching the surface, so to speak to a principle about how to approach this. I think that, and this is double-edged sword, because human tastes are also faulty, right? We like sugar, we like fat, we like salt, but I think we listen to our bodies. I think there's a way that if you are in tune with your body, you can kind of understand what's good for you. And it's not the same. Diet's gonna be good for everyone. There's a lot of complexity in that, but I think greater just body awareness, and this is like a personal thing is something that we need to get in tune with ourselves. We need to stop ignoring ourselves and externalizing ourselves with technology. John: You were saying like, oh, well we have these kind of desires for these foods that aren't good for us, but that is quite different from listening to your body, right? I mean, so I might want a McDonald's burger or something, but then I might listen to my body later and my body might be telling me, oh, like, that was like, not what we really needed. And I'm sure having one, one a week or something probably is [00:42:00] fine, right? But it's like, your body's different from your urges. Bryan: I'm sure that Crohn's disease is an illustration of that all the time. Your urges of what tastes good versus your experience of actually dealing with it. Peter: Yeah. Yeah. I mean, there's a particular pathology that means essentially if you're dealing with an extreme imbalance I'm not gonna say everybody needs to avoid grains. That's, that doesn't seem to be the case. But I guess there's the question of like, the Irish are sort of inseparable from the idea of potatoes or, you know, Italians from tomato sauce. I mean, those are no new world foods. I mean, it's maybe a little bit difficult even to really ascertain exactly what diet was, in a truly ancient fashion. And maybe a little bit of a limitation of applying the, essentially like eating what, four generations back ate is like, maybe some foods, right? Like a lot of cured meats, right? You know, we understand may have a cancer tie-in which, you know, given that life expectancy, has only, recently risen above like 40, right. You know, worked for them obviously up to 40 at least. And then you say, okay, well, you know, maybe cancer wasn't a problem that overtook them. So they did fine with that. So [00:43:00] there's some limitations of knowledge there. I mean, empirically speaking, you can look at places like Blue Zones and how they eat. and then it's like, okay, at least a diet is a component of what they're eating. Or at least, if their diet was that bad, it would be killing them sooner. Places that, oh, these places where people live, I don't remember. Essentially above 90 or something. But like Japan is a blue zone and there's, different ways of studying these diets where there's a lot of fresh food, a lot of natural food and so on. So yeah, if the food was that bad, it would be killing them sooner. So, you can empirically conclude that food is working pretty well for those people in those places. Bryan: I guess thematically it seems like a little bit of pushback against an effort to be reductionist about that, you know, a pushback against the intellectual hubris that we understand it all and we just can chop it up into these basic components and more empirically driven by traditional, traditional foods that seem to be working well for populations of people that are healthy. Chris: I mean, you know, recipes, some processes, communal traditions, developed over hundreds or thousands of years and [00:44:00] cooking particular foods to remove the toxins just by breaking down, or disrupting the proteins that are toxic. And they don't know why, but we iterated to a point that found something that worked. So yeah, I think the important thing is exactly what you're saying is like there's sort of, um, an then empiricism that we just need to have. I mean, ultimately what are we looking for? We're looking for outcomes anyway. So let's start with the outcome. Let's not start with breaking it down into a reductionist sort of framework. Peter: Yeah. You mean reductionism obviously has had a lot of, benefits and gains right? To, to the western world and beyond. I certainly observed that today though in the sense of particularly in a place like San Francisco, right? People are, very eager to hop on this sort of idea of scientific bandwagon, you know, where everything is scientific and it's like, but you see that this is very noisy, right? And it's one day we think one thing's true, the next, it's not true, right? I mean, science as a way of learning has its limits. Maybe an under-appreciation for, essentially the wisdom that's been accumulated up to this point, basically by [00:45:00] humanity Bryan: I'm curious to kind of switch gears a little bit, how you both joined SPC at separate times for both members here. How did y'all find unity on this project? How did you come to find yourselves collaborating? Chris: As soon as I got here, I was putting myself in front of everyone I could and just hearing out ideas and sharing my own ideas and I've had some theses about where the world is going. You know, I have some concerns about recession and about global supply chains, and the longevity of the global economic framework that we have lived in for 30 years now maybe depends on how you take a look at it, but 50 years. And so I saw this thought that I should focus on something that's really a base base economic activity, food, agriculture, transportation, and look at things that can be solved there and improved there because we're never gonna not need those. And so that was my inspiration. As soon as I encountered Peter and he had this focus on food and food preparation, I'm like, this is something that I can really get behind that I see a lot of value in for a lot of [00:46:00] people. I think people are struggling with the economics of food today, and I think it could get worse talking about eggs earlier, you know, I just want to see people be able to meet their basic needs. Peter: Yeah. So yeah, Chris and I started talking and realized pretty quickly that we were aligned in, in terms of a lot of our values and goals and just, you know, having a lot of fun talking about different topics. Yeah so we started doing the, you know, kind of co-founder dating process. I think we used some of the materials from whichever SBC alum, put together things like areas of responsibility matrix and, things like that just to help think through essentially if, we were the right fit to work together. And yeah. Bryan: So, we like to wrap it up with a recommendation. So I feel like we've mentioned a few things, but is there anything sort of top of mind to you that if anyone was listening that you'd recommend they'd go read or watch or listen to anything that's been on your brain recently? Chris: Well, something to listen to good music. There's a great album that I've been riffing on for about a year now. It's this very unknown album, prophet. It came out I think 2018. [00:47:00] But, it's got this song called Party on there. I want to be your man. It's just this really weird crossover of like eighties r and b and modern, electronic music that just makes me want to dance. Uh, . Peter: Cool, cool. Nice. Yeah. So something that Chris and I watched recently relating to the food thing too. We watched a documentary that somebody had put together on the auto map which was actually largely a significant inspiration for actually one of the first companies in the kind of food automation renaissance thing. Um, itsa. I was trying to do at least a format that was somewhat like that, although the cuisine was very different. But anyway, it was very interesting to watch in the sense of it was, this restaurant chain was around for nearly a hundred years. It served a huge fraction of the New York City and Philadelphia population. It was the largest restaurant entity for a very long time by volume. I guess some of the things that resonated with us were I think like the, just the really, the fundamental ness of treating people well, of making good food of figuring out a good strategy to essentially do [00:48:00] good, you know, by all the stakeholders involved. And just essentially how long they prospered as a result of taking care of people. Bryan: Thanks for the recommendations. it's been a pleasure chatting with you all today. Chris: Oh, the pleasure's been ours. Really enjoyed this, so thank you. Peter: Yeah, thanks. John: Thanks so much. I learned a lot. Bryan: Me too. That's all for Pioneer Park this time. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit pioneerpark.substack.com (https://pioneerpark.substack.com?utm_medium=podcast&utm_campaign=CTA_1) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

11. 

    23 Feb 2023

    Unstructured play and personal tutors with Cinjon Resnick

    John and Bryan interview Cinjon Resnick, an AI researcher at NYU. Cinjon is interested in developing unstructured, discovery-oriented play games and experiences. Topics * Ender’s Game and its inspiring concept of the Mind Game as an education system that deeply understands the state of the student. * The present and future of AI-driven tutors (see Cinjon’s longer post here (https://cinjon.com/building-primer)). * Virtues of personal tutors has inspiration back to Socrates and Rousseau, AI may be able to make this broadly accessible. * Currently LingoStar (https://www.lingostar.ai/) and Speak (https://www.speak.com/) offer digital tutors for language * These solutions still lack the empathy of a real human tutor, but that should be possible * This is a potential solution to Bloom’s 2 Sigma Problem (https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem). * Cinjon’s experience with circus, and his interactions with instructors including Victor Fomine (http://www.circopedia.org/Victor_Fomine) * The inspiration for Cinjon’s hackathon project, Animate. * Cinjon’s love of social deception games, and the challenges for developing algorithms that can effectively play those sorts of games with humans. * Impact of Generative AI on gaming * Motion Generation models like PhysDiff (https://nvlabs.github.io/PhysDiff/) or the Motion Diffusion Model (https://arxiv.org/abs/2209.14916) * AIs that are able to imitate human players * Advances in Chess AIs * Natural language APIs * Meeting the challenges of hallucination * The limits and potential of AI-driven storytelling * Why voice processing, is critical to build machines that have human-like empathy * The importance of real time processing, for example simultaneous translation. * The future of Multi agent RL models * Causality Research as an underrated field * Cinjon’s recommendations * Diamond Age by Neal Stephenson [goodreads (https://www.goodreads.com/en/book/show/827)], Ender’s Game by Orson Scott Card [goodreads (https://www.goodreads.com/book/show/375802.Ender_s_Game?from_search=true&from_srp=true&qid=WmfgBLOOcQ&rank=1)] * Rousseau on education: Emile [goodreads (https://www.goodreads.com/book/show/326679.Emile_or_On_Education?ref=nav_sb_ss_4_5)] * Martin Arjovsky’s thesis [arxiv (https://arxiv.org/abs/2103.02667)] * What’s next for Cinjon? He is starting to think deeply about childhood companions. Transcript [00:00:00] Hi, I'm Bryan and I'm John. And we are hosting the Pioneer Park Podcast where we bring you in-depth conversations with some of the most innovative and forward-thinking creators, technologists, and intellectuals. We're here to share our passion for exploring the cutting edge of creativity and technology. And we're excited to bring you along on the journey. Tune in for thought-provoking conversations with some of the brightest minds of Silicon Valley and beyond. Bryan Davis: Hey there. Welcome to the Pioneer Park Podcast. Today we are having an interview with Dr. Cinjon Resnick, who likes to be at the forefront of tech and research. He has worked at Startups, spent time as a fellow at Google Brain, and recently wrapped up a PhD in machine learning from nyu. He's an alum of South Park Commons and is currently working on ideas related to AI powered experiences, games and companionship. Cinjon, welcome. Cinjon Resnick: Thank you. Appreciate it guys. Bryan Davis: So I'm I wanna dive right in. Tell me about Enders Game, your relationship with the book Enders Game and the cool game that was featured there. I [00:01:00] guess that's called Mind Game in the book. What is your memory of reading that and what inspired you about that narrative? Cinjon Resnick: Ender's game is wonderful. If you haven't read it, I highly recommend it. It's a story about a a whole family. And this family is, they're special, but in particular, this one boy ends up taking on the mantle of savior of humanity through a simulated adventure experience. I'll leave at that and let the audience go and read it. But in particular, inside of Ender's Game, there is a story. And the story is guided by this thing called Mind Game. In this story undergoes on an adventure. It's not meant to be beaten. This is meant to test different aspects of the character. There's some fun things that happened, but this was a companion friend. There was interactive companion. It was similar to a diamond age where you have this concept of the primer. The primer is an interactive like Socrates for the child. . So we're dealing Bryan Davis: with a, now a set of [00:02:00] technologies that are potentially opening up the doors towards unstructured, discovery oriented play games experiences. And it sounds like that's a lot of the areas that you're interested in developing your work. What do you think are the experiences that are opened up by the current generative AI technologies? Cinjon Resnick: Yeah, it's a good question cuz if you think about what life was like for a 1700s, 1800s child either you had nothing or you were very wealthy and then you had teachers, those teachers were 1 on 1 personal tutors. So Rousseau talked about this in Émile, and like this idea of. A personal tutor that you would have for you. And so Socrates was this famously for different wealthy patrons, ch children, wealthy patrons. What would be really cool is to provide this for every kid. Are we there yet? Probably not exactly in this document that I wrote sort of goes through where exactly we need to fix and what are the remaining problems to be being met. But we're pretty close and we're close and close to each day. And so the opportunity now [00:03:00] to gear an an ai, if you will, towards a child's ability towards a child so that it is a personalized experience for them and what they want to learn and what they wanna go, but stories they wanna play to, to play out for themselves and really have that AI be something that is not just comfortable for, not just a great experience for the child, but something that parents want. We're close. We're close. And so I think ti that today is time to really start thinking about that and maybe even building towards it, finding some initial wedges. , are there some John McDonnell: kind of maybe early applications or milestones on that journey that you're the most excited about that we could potentially be close to delivering? Cinjon Resnick: Yes. So I think that one very clear one is actually everything that's happening, language learning. We see companies like LingoStar, I think Speak is gonna go this way as well, where they're developing applications that you could imagine just plugging a child into or yourself as an adult language learner and figuring out, okay, this is how you understand French in the context of actually talking about it with.[00:04:00] And from there, the interactive experience actually looks pretty similar to the interactive experience when you're language learning with a human. The thing that, the gap that's missing there, one of, one of the gap, there's two things actually that's pretty clearly missing. One is the empathy to know where the student is on their journey. And also on a day-to-day how they're feeling, how they're doing, do I up the capabilities of this AI or decrease them, et cetera. And the second one is just having a curriculum for the students because it's not just this interactive experience, especially with adult language learners, there's also this idea of at some point you are teaching them, you have some objective in mind for where they're trying to go. That's missing. But I think that we're in a path to build, able to get there and Speak, they already have tons of users in I believe South Korea LingoStar, they're trying the same idea. They're of demonstrating the capabilities of this. Bryan Davis: Have you heard of Bloom's two Sigma? Yeah. Yeah. Which for anyone listening, that's the idea that Language learning or [00:05:00] any kind of learning that is taking place with a single tutor. So the the impact of having a personal tutor on learning is Two Sigma greater than the sort of base case. Basically making the case for individualized tutors and individualized education and bloom's two Sigma is a pretty prominent result in educational theory. And I think speaks to the fact that these personalized Aristotles or these personalized coaches have a big impact on education. And so I suppose what you're proposing is that we might be at a point where we can make these personalized tutors and be achieving much more significant educational results for a much larger proportion of the population. Cinjon Resnick: That's right. That's right. So I have some background in some of this stuff. I've not been a teacher per se, but I do regular personal tutoring in, in circus. So I have a teacher in the circus apparatus that I trained, and it's just night and day, whether I'm working with one of my coaches or not. Similarly when we've tried to do things [00:06:00] around I ran a nonprofit called Depth First Learning for a while, which the whole goal there was to try and figure out a different way to learn from a structured base. I'm not gonna go into the exact details of how this is that person learning.com (http://learning.com), wanna check it out. But what was interesting there was when you put people in a group, rather than having 'em just be alone, it works out much better. Why is that? I think a big reason is because you get to learn from what other people are doing, or the people are going in where their knowledge is coming from. And so if you have a particular entity who's geared up to understand the topic better than you, and to be able to go on this journey with you, but is also tuned to knowing where you're at. . It's very powerful. And this is a lot of the things that you're pointing at, let's say with Bloom, et cetera. So one thing that's really cool about doing this with language learning is that it's largely just about talking. And so the subject matter is really easy. You can't, it's hard to get it wrong. A lot of these machines that'll hallucinate answers today, and they can get it wrong, say in history or in finance, that's a problem. [00:07:00] Another area where I think is very primed for this concept is in early childhood interactive experiences, because getting it wrong just doesn't matter to a kid that's, three to five to seven years old, getting something wrong in their story of their day or talking about, a, a big dog that is drooling in the park. And that doesn't matter if it's slightly. . But the experience of making a companion that I can actually have empathy with this child, those are the things you start to be able to build there. And that's actually a direction I've mostly been looking into. Bryan Davis: I'm curious if you've had any mentors or teachers that you feel have been really effective in your life in cultivating that experience for you, that's made you so interested in this. Cinjon Resnick: What exactly are asking if a teacher of mine has actually just effectively been a Bryan Davis: 1 on 1 tutor? Exactly. Whether or not you've had a really sort of significant relationship with somebody who is a mentor or a tutor that kind of really you felt opened up a new field. It sounds like to some degree circus has been that for you. Yeah. Cinjon Resnick: Yes. Yeah, I can definitely answer that way. So [00:08:00] in athletic movement I have two coaches in Montreal that I train with one, and this guy Victor Fomine, who's world famous coach, I'm really lucky to be able to work with him. Another guy think Sergey and I go to them for different things. , but the, and Victor also doesn't speak English, so the opportunity to work with them is just fantastic because they understand home so much of how the body should move. And so the whole experience with that is getting cues, getting, figuring out, okay, this, so we should be doing then, so we're doing then. And so just the cue to tap Tapi, look at your, look at the ground, look at your feet. Just hearing that over and over again at the right time is fantastic. It's just so useful. But then sometimes I can go there and I'll see him train, I'll see him teaching people who are much less skill or even much higher skill, and he changes his course to those people, right? There's an empathy for understanding where they're at. But then still this drive to I think one of the best parts of working with a tutor who's able to adapt to you is if you give more, then they give more.[00:09:00] And if it's a day where you just can't give that much, they recognize that. Being able to build that into this next generation of machines is gonna be so important for getting this tutoring experience. Bryan Davis: One last aside on that, what is your circus skill? I do Cinjon Resnick: straps. It's like artistic rings. John McDonnell: What was the inspiration for your recent hackathon project that you did at South Park Commons? Cinjon Resnick: Animate. You're talking about animate and the idea here, just to sum it up for the audience is we're going to. , I wanted to understand what was the state of the art in a wide variety of systems. There's a wide variety of APIs that we could use to have an interactive experience with ai. And additionally, I also wanted to understand what it would be like to do test. Two things. One is it fun to be read a story to? And two, is it interesting for a language learner to be read a story in different language in like the language that they're [00:10:00] thinking about? At this point, I hadn't yet come across any app that could do the second thing I have in the time since. But the Animate then was we took a chapter of Alice and Wonderland, we turned it into a visual story so you could see a scene with it, and there's a narrator, there's two characters. And then we wanted to have for each of those characters, them talking out their role. So in other words, we turned it into a play. Yeah. And the whole experience of taking the story, turning it into a play, and then animating the play so that you have the characters with their, their their mouths are moving. Then the, oh yeah, then there's the language switching and the interactive experience. So the main goal was really to test, is it interesting to be read a story to what would make it interesting and is there something around language learning there that can be, that can tap into, so built all that out. It was actually rather quick to build it all out. Considering the technology today is very good. Just progress to the point where [00:11:00] you can do all these things and. The goal at the end of this was then put it before some children and see what they liked. And when I did that, there was just one thing that stood out over and over again was the ability to change the scene. It's just fun. It's a fun experience when you edit the scene and you go from something which you can see, it's plain, and it's a canvas. It's a creatively constrained canvas because it's it's characters sitting in front of a fireplace with a chessboard in between them. But then you say, oh, I wanna put a monkey on the chessboard, or I want to change this chair to be a giraffe. And what you get back is just fun. It's surprising, it's creative. It's interesting. It's hilarious. And it was engaging. Watching the kids actually Bryan Davis: play with this. I'm curious, you are very interested in games and play. Do you play any Cinjon Resnick: games? I do. I really like social deception games. So famously like secret Hitler and coup those kinds of games. But I also braid and I used to play a lot of Diablo too when I was a kid and [00:12:00] work off three and those kinda things. Bryan Davis: Do you spend any of your time now deeply invested in any kind of computer worlds or most of them are sort of social deception? Ooh, . Cinjon Resnick: I don't really play any computer games these days. I am gonna play double four when it comes out. I have a childhood nostalgia around it, but I have not invested in any of the ones that I've noticed in my friends playing. I'm not, I never got into factorial, for example. , do you feel like Bryan Davis: the next generation of games? Factorial is a great example of a game that is algorithmically generated, but it's and has a lot of ran randomness that's embedded in the way the game is played. It's very famous for being replayable and time again, every experience will be different. Do you think that generative ai, and I guess, how do you think the generative AI is poised to impact gaming? . Cinjon Resnick: That's a good question. So I think there's some obvious answers here. Things you can point out with AI Dungeon, you could talk about storytelling, whatnot. I think there's two things that may be are less obvious. One direction is around motion generation. So we're starting [00:13:00] to see this past year, really actually this past year, motion generation start to be, start to work. What I mean by this is examples are PhysDiff or the motion diffusion model. These are pointed to a place where you can just say, I want this character to move like Beyonce in a rainstorm with jazz music in the background. And then it, it does some interesting thing because it has some concept of Beyonce rainstorms and jazz music associated with the movement of the human body. , we're not at a point where we can do this with. With shapes that aren't really the human body unless you can slap a faux human body on it. But what that means is that if I was to just draw something that had some resemblance to the human pose, you could imagine creating, turning that thing into its own shape. And so this has a, this could have a huge effect on U G C content. Suddenly U G C content can come alive. So I've seen a couple of startups working along those directions. Not the direction of taking the [00:14:00] motion generation and putting in yet, but being ready for when that's possible. So that's one area. Another area which I think is really promising and is, I know is actually being worked on in place like EA is defining difficulty differently. A lot of times in game difficulty, what is you see a a computer get better or stronger. just based off of they'll give a bonus as to how much gold it collects when it accumulates something, it's just hacks. . But if instead what you can do is define it in terms of how capable it is as an agent and that capability adapts to what your strengths are, so you're doing really well. So they'll just keep upping the difficulty until you're in that sweet spot of just a little bit past what you can do. But if you strive hard, you'll get there. I think that's gonna come, that's gonna come pretty soon. Yeah. So like John McDonnell: I play online chess and [00:15:00] I think the thing that is very cool about it is that you have your ELO rating and you're always paired with people who are a good match. And it was always disappointing. Like I remember as a kid when I realized that like in civilization, deity mode or whatever, like all that was happening was like the bots just. Could build every building in three turns or something. . I was like really disappointed. Like I thought, oh, the bot's gonna be like super intelligent. It's gonna outsmart me. And that's not actually how it works. And you just kinda have to figure out what hacks you can do to exploit its dumb behavior. Yeah, it's like a completely different idea to actually make it be really smarter. Cinjon Resnick: Yeah. Yeah. I think places you'll see this first are things like FPSs. In chess you could do this right now, you could train an alpha go to have any ELO rating you want. Yeah. And then just park it on the server and have it be available to play, I yeah. I dunno if anyone's actually done that though. actually John McDonnell: think so. So I for sure heard about people building chess bots that are intended to have a certain ELO rating. And then I believe actually that on chess.com (http://chess.com), some of those bots like I like are actually Yeah. Spec specifically trained to behave like a human would behave. I if the human had that ELO Cinjon Resnick: rating yeah. Bryan Davis: I think some go servers also have [00:16:00] similar bots that are out there at different levels with different sort of training and background. So very interesting to think about. I recently read the paper about diplomacy, the Cicero paper from Facebook, which was talking about the integration of large language models into this sort of like social strategic game. And that was one of the most fascinating examples and that I've run into in recent history of an integration of a very complex social game with a strategic engine. I'm very fascinated to think about what is the sort of next version of this. Are there environments where we could let these things loose so in learn from it proactively. A lot of these things, especially the strategic engines that the Alpha go and these other sort of game engines that are winning these strategy games rely on self play. Hundreds and hundreds of games to be iterated upon, in the background playing each other. And I'm curious, do you think that in a social game or a game that is almost dependent on relationships with humans, do we run into an issue where self play becomes ineffective [00:17:00] because we can't actually mimic human behavior, we can't mimic human Cinjon Resnick: adaptability. What is the world where, what is the game you're thinking of where you need to do that? And guess I'll point out my example is being in AlphaGo, once it passed human capability, they kept getting better because it was now competing against its own population. So the Bryan Davis: example of diplomacy, I think is somewhat interesting because it is reliant on human communication. It's reliant on interpretation and alliances being effectively formed. . And so perhaps there's a category of games that do have this sort of like unbounded social nature and self play. When they used self play in the context of diplomacy they found that the a large PORs, a large part of diplomacy, first of all, takes place over messaging, basically convincing people to ally with you or to invade another country on your behalf. And so that requires that you're able to be persuasive. And when they instituted self play in this system, they found that there was a tremendous amount of semantic drift, where system one and system two were [00:18:00] communicating with each other and they were be beginning to use nonsensical language to And so that seems to be a limiting factor on how well a computer can do in a sort of social setting or a setting where a computer needs to be persuasive. It seems like there needs to be some sort of anchor to the real world. Cinjon Resnick: Yeah. Yeah. So it's been a while since I've been involved in this research direction. I would say the thing that come to mind is called other play. If you haven't seen that, I would look into that. So other play is its work out of I think it was also actually originally a fair, but I associate it more with Jakob Foerster and his lab. The idea behind other play is that you want to train agents that can work not just with their self, but with other agents. And so the goal the whole time was to be able to train agents that play Hanabi with humans at a very high level. And so the algorithms that they come up with around this, even though they're playing self play, Need to [00:19:00] be able to work with humans too. And they actually do a pretty darn good job. So a lot of that I think that there's a lot of room for algorithm improvement where you go in those directions. The challenging part is always going to be to keep the human connection available there. I think though, there's another question that's built into what you're seeing, which is can you make an algorithm that doesn't actually work with humans, that is agents getting better and better, but still a human interpretable as to what they're talking about, right? So that that's, it doesn't need necessarily need to be playing with humans, but needs to be talking in a way that humans can understand or that's what we would want. And one question here is if it's even possible, because maybe what they're saying looks interpretable to humans, but actually has codes underneath all of it. And so that's, that's I think an open question. I don't actually know a research that has addressed any of that but I would expect that to actually happen that once it surpasses the human, trying to understand the strategy [00:20:00] involved is too difficult. And at some point it's going to be so difficult that we're just going to let it happen anyways. We're gonna let it happen because the results are so good. And you can take that as for voting, even I'm not. . Bryan Davis: One of John, one of my favorite conjectures that John has about the future is this world in which there are just natural language APIs to the universe. So basically, every sort of site or service has a natural language API where you state your intent and it is able to perform the actions. And you can have, obviously these APIs that are beginning to interact with each other, just, like a large API server, but they're interacting with natural language. But what I think is interesting in that context is what happens when natural language ceases to mean what it means to us when these bots are that's right. Yeah. To communicate in their own version of our language that to mean very different things. Yeah. Cinjon Resnick: But I love that direction, that emerging communication and as, one of the reasons I wanted to do a PhD was to study that area. And I think that there is, , there's [00:21:00] a lot of fanciful things that we can come up with in that domain. It's just, it's hard to then ground it in a real ac actually useful thing to do. And you saw, we've seen now the rise of agents that we can talk with. And we used to call these, three years ago, we used to call these chat bot, and now we don't, we call them, we just, we've forgotten the word chat bott. Instead we go Just ChatGPT or GPT-3 or the coming one from Google, whatever. But they work now and they work in a way that eschewed all of the purpose that was going on with emerging communication, but maybe it's time to bring it back in. And I would love that. I think that'd be amazing. I also wanna bring up something else I think is interesting in this direction. And that's that's how it's sort of connection with hallucination the hallucination in terms of these the big language models. So my friend Colin has this interesting take , he says it's, bzip, it's pretty hard to imagine these neural nets being more compressive than bzip. And bzip is roughly N over four in terms of using float 32. So if you have size of your language, just [00:22:00] 25% in, that's roughly bzip. Let's just say that's a floor. Okay? So then let's just, we're gonna move on from what Colin's point is there, but that's our floor. Now, if you imagine all of the internet that's generated, it's much larger than the size of these models. So if you imagine stuffing all of that into these models, it's not gonna be able to. in the same way that you and I, when we go around the world, we can't stuff everything into our head. We have to compress a lot. We have to figure out how to make it compositional, but it's not gonna get below what bzip is doing because it's not even caring about making. You can do it John McDonnell: if you're lossy, right? Cinjon Resnick: Yes, that's exactly it. You can do it if you're lossy, and that's where hallucination comes in, because we don't give it any faculty for knowing what it doesn't for knowing that it doesn't know something, and we require that it generates something. The only answer is that it's a lossy hallucination. It has to be. And if you were to instead figure out some faculty for either having a reliable communication channel that [00:23:00] let's it say I don't know what this is. If you don't even do that, you're not gonna get, if you, sorry, if you don't do that, you're not gonna fix this problem. Bryan Davis: So what do you see as solutions to hallucination in the, short and medium term? . Cinjon Resnick: I think the first question is to ask yourself if you need a solution. Cause a lot of times maybe you don't need a solution. Yes, you're gonna need one. If you're trying to do something that's legal obligation. If you require that this thing is airtight in that domain, then you need a solution. But in many places you don't. And ask yourself really, if you do the second answer is at some point we need to teach it or it needs to emerge because that's the seems to be the flavor of the day, is to emerge a property of understanding what it doesn't know. And there are places that people working on that, but even the direction those say involving knowledge bases inside these things, know, people have been doing this for a while. It's not like in the last year and a half was the first time that we started to understand that the stuff can hallucinate people working on summarization for decades, extractive or subtracted summarization. This is not a. [00:24:00] A new topic. We don't have an answer even if you include knowledge base because the network may actually have a concept of the knowledge base, sorry. The model may have a concept that this knowledge base exists without being able to actually point to the fact that caused it to understand something. And in other words, I don't think there is a solution right now. And I think you have to just deal with how much you wanted of it and then otherwise form the right gates you form the the right playpen for your users or whatever to plan. John McDonnell: Yeah, it's really it's interesting your point about the fact that I can't say that it doesn't know like it makes me wonder, if you could instruct tune them or something to be able to ask the follow up question. How confident or were you sure about that and to have it reliably, give you a reasonable response or that kind of thing. Cinjon Resnick: It's unclear though that would help, right? Because it's the same problem. It's like at its core, it just doesn't have the ability to do this again, unless something emerges. That's different. But we haven't seen that. Instead, what it has I've seen really good evidence to suggest that [00:25:00] what it has figured out is the ability to follow your intent. The conversational partner's intent. . So if Brian's talking to this agent and it knows what it's looking for, is some answer along these lines, like why do you not know what you know? What do you mean what I don't? Is it because you are trying to track this fact? Oh yes. That's why. And then you roll dice again and says, oh no, that's not why it's actually this. But the understanding, the, it has, it seems to have some understanding of your intent to where it's going with it. Bryan Davis: Yeah. That's interesting when I phrase this is they seem to be very agreeable and I, yeah. Yes. I suppose that's because a lot of the to your point earlier, like these are trained on data that exists, not data of just denial of existence. And I think it's interesting, it's perhaps an interesting point to think about. The negative case of not knowing is not represented very well in the data that it's trained on. Because the [00:26:00] overwhelmingly the internet is full of information. Even if that information is false, it's not full of people. Or I guess we have an underrepresentation of questions not being. because the questions that it's being trained on are content. So it's almost as if we have this bias towards the things obviously that do exist, the training sample that do exist, and perhaps there's a, there would be a benefit to generating false or negations as part of its, as part of its training sample. One other strategy in this domain that I'm curious to hear feedback on is relatively annotation heavy. And that would require basically taking the input of something like Wikipedia and annotating it as requiring citation or being basically, Labeling as this particular statement coming from a, needing a source or being an example of something that is a timely, factual piece of information and thereby [00:27:00] perhaps teaching a model in its process of training that it needs to basically inject some citation or inject some sort of timely fact. And knowing that and being able to output that as part of its response to then be filled in by so we can imagine, for instance, a tag that indicates a timely fact or a sort of citation needed that's actually in the data as it's ingested. That's just one strategy to throw out there. I understand that it might underlie some of the experimentations with the FLA model from Google. But curious if you have any reactions to that sort of strategy or Cinjon Resnick: others. I think that it's a great strategy for targeting it towards your use case. , if you care a lot about having, lemme put it this way, the model size isn't changing, so you still have this limitation this more meta limitation around can you actually put all the data you need into [00:28:00] this thing? It's comparable to you as a human. Actually. It even has fewer parameters and abilities right now than you as a human. But at some point it'll be comparable in terms of per of parameters in its head. And you yourself can't remember everything that happened. There's just too much data out there. And so the answer must be that it has to compress it into composition ways and then use those compositions to, to meld into these new concepts and then we'll explain them. But even us, when we do that, we still don't remember facts. Because those are too, there's too much information there. And it's too much long. It's too long. Tail. What? I don't anticipate that changing it. What do you see? Why would it change? Wow. Bryan Davis: Sorry. What do you see as the limits of AI driven storytelling? What's the boundary in its capacity to proactively create? Cinjon Resnick: That's interesting. I think in a long term, I don't think that there is [00:29:00] bound it's not bounded. I think that it'll gain all the faculties that we want it to have. I think today, one way that it's, one way that it's bounded is definitely in the empathy and understanding of what's going on. If you try and say, okay you're playing the role of a teacher for a five year old, It's not gonna remember the entire time that's playing with a five year old child. That's one thing that comes up. But at some point the child's gonna say some set of information and the model it's not gonna know that tone means something. There's no ability to take that in. If you say the kid was excitedly saying it, are you saying it with the right way that we gauge it? There's just, there's a lot of lossy information there in how humans receive empathy and give empathy to get where the child is at. And I expect that'll be true for us as well when I've taken. We put that back to adult language learners. When I've taken language classes [00:30:00] or just one-on-one experiences with teachers they have this ability to slow down the way that they speak automatically to gauge where when they figure out that you're not thinking about the right thing, or they can stop and say, oh, you didn't get that word did you? Or on the other flip side, they can speed up when they recognize, oh, you're just, you're fine. You got this. Let's go faster. It's not gonna be able to do that automatically. So there's gonna be this little bit of extra friction every time you use it, where you need to now account for that with design. . And I think it's possible. And it's a very interesting journey in the next 10 years, getting from here to the next step. John McDonnell: How would you try to get it there? Cinjon Resnick: The answers that come to mind feel like a combination of getting the data and getting the right design today. And also just we need to reduce the latency in things like speech conversations. So in a past life, I've worked a lot with audio data. It's, we're talking if it's 16 fpf, oh, sorry, 16 kilohertz, then you're talking tremendous amount of samples per second models today can deal with that. [00:31:00] But it's a whole other modality compared to text, which is much fewer samples per second because each of those samples, much fewer words per second, because each of them contain a lot more information . So if you wanna be able to go from what the experience that we are having right now where I'm talking, and you immediately understand it because there's no extra steps from taking this audio to text, to, to sensory, to, to reasoning, and then reasoning back out to text. That pathway needs to be smoothed. And there is some really interesting work going from audio to audio. But most of the big labs are not focused on that because they're seeing so much power right now. Go into these straight up text to text models that they're gonna focus on them for a while. If you wanna get to a place where it feels like realtime understanding and realtime maybe even empathy it's possible that emerges from the text, but it's gonna do it in a medium that doesn't feel the same as it does with you and I right now, or what happens when you play with a child. And so I, I do believe you, you [00:32:00] almost surely have to go to an audio to audio experience to get there. John McDonnell: it feels like there's almost like a multimodality to this where you can think of it there as being the text itself is like one mode, and then all the kind of like meta text, audio information about the way the person's talking or their speed of speech or their accent or whatever. Is this other stream that, that you're actually gonna want to co-pro as you're making judgements. Cinjon Resnick: I agree. Yeah, I agree. I think that there's so much interesting questions around ity that we don't really understand, and sure there's dire other fields that look at the effect that comes with different pro, but bringing that into the end-to-end experience that is, that's in advances today. We, we don't have good answers yet. There are teams working on this. There are teams at Google or Facebook that I'm familiar with that they're not they're, they're not even private about it. They're public. They're fairly public about the fact they're trying this because it's all early research. John McDonnell: So Brian and I, for our hackathon project, we made a, like a voice [00:33:00] chat bot that you could call on the phone. That's awesome. And it was what, honestly, I think the coolest thing about this project was having that experience of talking to a bot by a voice and then seeing how it's cool and also how it's broken. . And so like when we first turned it on, we used curate as the model and the response time was about a couple hundred milliseconds. And it really felt like Curie is like listening to us and then answering back. And it is really magical just to have the bot be talking in a conversation flow and cadence that matches yours at least a little bit. But then curri, curri was difficult because curi hallucinates a lot. So fun. It's fun that Curie hallucinates actually, but it was like, . I had weird conversations about it where it told me there was like a terrorist attack going on and stuff, and it's OK, , I can't really ship that. . But it was real. It was, and it was creepy. Like it was almost like just there's like a freakiness to that. But but then, so then to get reliability switched to Da Vinci and then it was like, three second lag time or something, and then you just kinda feel like you're giving instructions to [00:34:00] Alexa or something. Yeah. Magical. And it's, and of course like mean to your point man, if you could get that curious speed and then also have the bot be attentive to your ity and the kind of like other aspects of your speech that, that are reflecting your state of mind. Like I could imagine that even if it was just not very smart, like being really magical feeling, Cinjon Resnick: yeah. It really would. And then, I that doesn't even count for the TTS and assr on the other side. So the Texas speech and the speech recognition on both sides of that, yeah, that's probably adds another few hundred milliseconds each way at least. Yeah. Me. This is, Bryan Davis: I think, evidence of how amazing it is to be a social animal and to have a brain that is capable of interacting in a real time, interactive, perceiving, understanding, reacting, all happening so quickly with and it speaks to the fact that there's some amazing compression representations of our world are extremely efficient and in their [00:35:00] ability, both in terms of a memory standpoint and also in terms of a computation standpoint. And I wonder whether or. The, that seems to me from from where I'm standing to be, one of the main limitations that of our current sort of understanding of how AI will progress is we are very far away from being able to represent the world in an efficient way that will allow for real-time communication, realtime speech, realtime video, that sort of thing. Yeah. Cinjon Resnick: You're right, and it's tempting to make predictions that this is very far away, as we've seen things move fairly fast sometimes the stuff that's coming out with respect to music is really incredible, but it's also not realtime. And the, maybe one benchmark to consider here is whether you can do simultaneous translation. People care about having simultaneous transla. , there's big companies that care about it because it means you don't have to take translators with you. There's large organizations that care about it because the UN then can be just having a much more efficient experience [00:36:00] on their floor. But wow, is that a hard problem? The idea that I can be talking right now and that there's someone right to my, in terms of the order here, maybe just someone right below me who can be with only half a second, maybe delay or a second delay, be translating the concepts that I'm saying. It's extraordinarily different than what machine translation does. Machine translation is going trying to do almost the sentence by sentence experience, but here it's more the conceptual experience in order to make that fast enough. And we do not have any good solutions for this. And I think this is probably akin to all of the problems that we've described here with respect to understanding empathy, et. Bryan Davis: what do you believe are the constraints on solving that problem? Do you think it's a understanding of art model architecture? Is it a a hardware issue? Do you feel like any of these things will be breakthrough points? Cinjon Resnick: I think it's largely data. We just don't have, we have tremendous amounts of data for doing machine [00:37:00] translation, for doing simultaneous translation. We have un which UN data might actually be around this this kind of good stuff. I don't know how much Bryan Davis: tens of thousands of hours of recorded un simultaneous translations. For context here, I used to be a translator. I was never a sim. I was never a s I was never a simultaneous translator, but I was a I would. What was, what's the other variety? I can't recall. But basically taking part in meetings with lawyers and translating back and forth between lawyers and clients. And this was a profession I was pursuing and I was fairly close with some people who did become simultaneous translators. And it's, it is amazing to think of the sort of computational training that they are enhancing specifically one part of their brain to be able to instantly code switch in their heads at the speed of human language. And it's very unique and it requires years of training to get right. Yeah. It's Cinjon Resnick: wild what happens in that when they're working on it themselves. Bryan Davis: It's like practice. It's like somebody trying to become a concert pianist is they [00:38:00] just perform and perform. And of course, they're also working to close gaps in any vocabulary that they might be missing. And become domain experts in the variety in the fields in which they really want to concentrate, whether that be politics or economics or specific business experience. So there's vocabulary acquisition that goes along with that training, but a lot of it is sitting in a booth and doing the work over and over. That's cool. Cinjon Resnick: Over, yeah. I really respect that a lot. Tens of thousands of hours. Sounds like enough. But I don't know. I don't know. I've not worked on the problem. I really, I haven't really thought about translation seriously as a research endeavor in four years, but I do perceive that it hasn't reached enough of a, it hasn't reached, it hasn't reached a place where people could say, Hey, this is almost ready to tip over. Let's now just add compute to it. There's no service that offers this. There's nothing That's good enough. John McDonnell: Switching gears a little bit, one thing that, that I really wanted to ask you about was this kind of world of multi-agent RL models. So you've done some work in this, right? , [00:39:00] so it's funny so now, in the Bay area, there's all this excitement about LLMs and everyone and open eyes, brand name is just like infinitely high. And of course they started off doing a lot of these multi-agent models. My, my impression of how this went was that they went transformed when transport, when the transformer paper came out. Then they built GPT and then they realized, oh, this is amazing. We're gonna just like pivot until we're really focusing on this. But I guess What's of become of that multi-agent work? What were they hoping to get out of that and did it just not work and are other people achieving their aims? Like how did that, like what's the state of that field? Cinjon Resnick: Yeah. Also a good question. It's, I don't know their motivations in particular. No. I will say that there was a long period. where people thought that the way to get to general intelligence was through RL, the reward function was the most important thing et cetera, et cetera. And you can learn everything through the reward function. [00:40:00] Theoretically, that remains true, but in practice it's appears to not be as important as having transplant data and a sim simple enough objective that still works for what you need and what the language models with respect to where multiagent stuff is happening. Fair is still doing quite a bit of it, as you can see with Cicero led by Noam and team. Then you have DeepMind, which of course has a bunch of people still work on this stuff all the time. I saw for all recently put out a paper that was really interesting, the 8 0 1, which is all about adaptive learning and be able to do it with small number of samples. , a lot of this work is now building on top of foundational models and then adding RL to it, which is what with RLHF as well. Sure. I think that for the near future it's going to a lot of that, the core multi-agent RL type stuff is gonna be relegated to academic [00:41:00] labs more. I don't know how much is gonna happen because everything is just super hot right now in working with foundation models and then pushing on that. Yeah, and there's also this feeling in academia that, more and more people do the thing that's hot. It's pretty common. And then every once in a while you're gonna have this offshoot that comes around that pushes things forward. It's gonna be surprising and there's gonna work, and then it's gonna take over a little bit more. There are labs I can point to that will continue on this path because they, it's not going to be run over by the computational steamroller so much. , there are important problems to think about that, that say around cooperation. like involving humans. I, if you, it's rare that you're going to be run away with a competition steam roller if you have to involve humans in the loop. Yeah. It's just too hard to then do it Now. Maybe r l HF will lead to some route where you can have people who are [00:42:00] every second are updating something, but you're gonna have to have a huge team doing that. And there's a bet here, say even like the Forester Lab in Oxford is kinda making a bet that actually this is going to continue to be the case and they're important problem to solve. Frankly, right now it just looks like all of research is being dominated by this stuff. And my old advisors at NYU certainly also are seeing that too. And every once in a while like something on Twitter or a comic or something about how all of ML research is now being taken over by these things. I You say that, I think there's one direction though, which is maybe answers your question a positively and that's around robotics. So Open doesn't really do robotics anymore. They stopped and they stopped because it's a different use of the resources that isn't gonna scale as well as this, as well as everything they're doing right now. Yeah. There are teams that are focusing on robotics. There's a, or team at robotic deep Mind, a team at brain. There's teams all over the world that are focusing on robotics still, and they have to bring together so many different parts of this, of the stack. So they're [00:43:00] starting to use LLMs in order to guide. The progress of the robot in order to make it do things that are human controllable. That's say, can, as an example sorry, the paper called Say Can, as an example. And they're also starting to do a lot more immense amounts of simulation. And using all that data and figuring out how to do that in a proper way. So we're gonna start seeing papers much more many more papers come out with high amount simulation and then doing a little bit of symptomal using the fact that these LLMs have so much understanding of the reel. . And that's really cool. That's happening as well. So I think that those are areas that, in terms of the multi-agent l I think you're gonna start seeing it seep into robotics more than you have because some of the other problems that they've tackled, their goal, sorry, that they've been focusing on will be easier to address. Given the lms. It does actually John McDonnell: Remind me of how, So I'm so old that so I had the opportunity to take Yann LeCun's class in like 2010. And I remember thinking like, [00:44:00] oh, like I know neural nets aren't cool. And he's just so obsessed with neural nets. I'm just gonna go take like a class that's like doing Pac learnability with SBS and stuff. And that's what I did. And huge regrets. It was obviously in retrospect to dumb decision . But, young really had to fight through and Ben Gio, these people who were, who kept working on neural nets, like they, they like like the field really abandoned that direction and like other stuff got trendier and they had to just say I'm just gonna work on this anyway. And at the end they were right. I do of wonder if a lot of this kind of multi-agent stuff or l like the trends going away, but like you can see what the potential is and some of the people who just really stick with it might end up being. Cinjon Resnick: That's causality today. Yeah. Oh my gosh. I think of causality as being that today that everyone has look, Facebook, just, Facebook just dropped their causality teams recently. It was part of the firing. There's, if you wanna have something that will push these things to the next level, it's having causal understanding. But we don't have real good ideas of how to bring causal [00:45:00] understanding into donuts. A lot of work on it. Cindy, you Bryan Davis: Take a moment. Uh, Can you define what causality is as a field of Cinjon Resnick: research? Oh, . Just ask me to define causality. What causality is a field? So causality is a field is trying. So I'll start by saying that there are really good conferences for causality. There's also, there's also a part of it, which is in fa fairness conference, it collides somewhat with that direction. And also I'll put a quick pitch here for Jonas Peter's work. He's amazing. Professor Zurich, who's been doing this stuff for a while. Christina Heinze-Deml Martin Arjovsky David Lopez-Paz these are really good researchers in AI. What you're looking for here is the ability to have some sort of the model to give the model a causal understanding of what it's doing. And there's some toy examples here you could throw out. One of 'em is if you have a data set that has it has a really weird distortion around, say women mostly have blonde hair, men mostly have brunette hair. [00:46:00] And then in your test set, it's flipped around. The models will tend to do a correlation there. And if they see something with blonde hair, they'll predict it's a male. Whereas what you really want is for it to have a causal understanding, or sorry, to have an understanding that is hair does not predict gender, or sorry, hair does not predict sex. And that ends up being, you could wait, think about this, is that there is, the causal link is broken there. So in terms of the graphical model, it would look different if that was predictable, if, sorry if sex was predictable by by hair color. So that's the toy problem that people oftentimes use for this. And you can even more toy just by using some Gaussian models to, and then making predictions about that. And we just really don't have good ways to scale this up, that exact toy problem. I can point to a solution. Martin's got a great one in his thesis of how to solve that one. But in terms of scaling it up to the full data sets, [00:47:00] real data, et cetera, et cetera, doesn't work. and going from A to B on this is really important if you want these things to actually have some sort of core understanding of what they're doing. So I think that there's this general hope right now in the field that when you go from 200 billion parameters to 200, maybe 2 trillion parameters, that it just solves it. It just happens, but there's no scientific reason to think that it's true. Interesting. And so I think a lot of this has been forgotten because the work is just, it's just working so right now. But and, sorry, but I forgotten. I just mean it's been put to the side. Bryan Davis: Yep. So what we're talking about here is perhaps an embedded notion of how the world works, perhaps an idea of internalized physics or understandings of the kind of structure of the environment in which it's being trained. rather than just [00:48:00] correlations about entities, which, to be fair, LLMs really seem amazing, but they are at their core really just predictions of next token. Cinjon Resnick: Yes. And so what you're pointing at is something a little different as well. What you're pointing at is having this embedded world model and being able to condition on some world model that's different than having a learn from data, causal understanding. There's different things we can point to and say, which is better, which is worse. The oral world will oftentimes say that actually what you want is these world models. That's not always true too. You're talking about model free versus model based here, but in terms of cause understanding, I think what you're saying is a great next step if we can get to the point where we can use these world models in a reliable way. A plus involving physics into things a plus. Awesome. But what we ultimately want is for it to be able to causally learn from data. And so when you go about the world as the human you can learn that this mirror sits upon this desk. If the desk moves outta the way the mirror will fall. It's not [00:49:00] clear at all that we bring into that experience any sense of this world model of physics. Instead, we have some just causal understanding that this desk is upholding this mirror. Is John McDonnell: it even conceptually, like this is actually, this seems like actually philosophically difficult, right? I, was it Hume who's who had that thing about how causality just can't be inferred from data? Cause you, you're really reading into your data. You're saying like, oh like I've seen this correlational structure before, but , really there's these kind of like rules underlying that and okay like the, and so some things that I see are because of the rules and some things that I see are due to rems in the environment, and I'm gonna go through and instead of decide like which things are which, and infer this like rule set, which I'm then gonna believe, but even my rule set might have problems. So I also have to have uncertainty by my rule set , right? Is that even conceptual? Like this almost seems like a philosophical Cinjon Resnick: problem. Like Bryan Davis: I guess to some degree it's also empirical because I do believe there's evidence that some sort of core understanding of physics is baked into our baked into the model, like pre-baked [00:50:00] into the model. So it's not being derived actively from interaction with the world. So it's very interesting to me to think about what elements of this are hardwired, in the circuitry and what elements of this are learned through interacting and giving that feedback from the world. Cinjon Resnick: . Yeah. These are the questions. And in respect to the philosophical thing is, another one you could ask is do we even need it to be causal or can you just have strong enough correlative things that actually ends up just being fine? Yeah. And there's no real issue. We don't have the answers to this. Yeah. And because we also don't have the answers to what humans are doing with this. Yeah. I have a suspicion that if you want to get to a place where you can have reliable answers come out of a model as to what it knows, what it doesn't know. You want it to have some underlying facility for doing this. Yeah. And perhaps that facility is not purposefully done. It's not built into as a prior in the model. Perhaps it's just emerges, but you need to be confident that it exists and we're It doesn't exist today. Yeah. [00:51:00] But the research into how it could exist, that's what I mean by the field of causality. Yes. That's a really interesting question. Bryan Davis: Perhaps we could wrap up with a question about a recommendation for listeners, something that you've read, or are reading or have been watching, or a game that you're interested in sharing. What would you, what's a takeaway from our conversation that you think you'd recommend to somebody? Cinjon Resnick: I think there's cool book conditions clearly things like Diamond Age Enders Game, which we discussed earlier. Another one along those lines, which I was reminded about from a friend recently is Russo's Take on education. Emile. Those are some pretty clear ones if you wanna think about this direction. I. Respect the causality stuff. Martin Arjovsky's thesis is fantastic. Very approachable. Yes, there's a lot of math in it, but if you want to ignore the maths, very approachable regardless. And [00:52:00] those John McDonnell: are great. Yeah. And I also just wanted to ask what's next for you? What do you want to build research next Cinjon Resnick: Yeah. It's it's fun. I'm right now figuring this out exactly where I'm gonna commit to, but I'm spending a lot of time thinking about childhood companion and how to use the modern tooling to really make something that can grow with a kit. If you can get, let's just imagine you have a child when they're five or six, you get them to love an experience, an interactive companion, and you grow with them over time. I, I think this will properly ride the wave of research such that the clean ones around understanding more memory, understanding more empathy. I think that what you would have from this is the ability to form a lifelong companion. If the kid form a lifelong companion is something that can really help them a lot. And and all the tooling is on. Its very cool. John McDonnell: I really love that idea. I feel like I want a lifelong companion.[00:53:00] Bryan Davis: Thanks so for being part of Pioneer Park. Cinjon Resnick: Pleasure. Thanks. Appreciate, John McDonnell: thanks so much. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit pioneerpark.substack.com (https://pioneerpark.substack.com?utm_medium=podcast&utm_campaign=CTA_1) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

12. 

    8 Feb 2023

    The frontiers of clinical AI with Vivek Natarajan

    Check out our interview with Vivek Natarajan, a member of South Park Commons and coauthor of the recent paper on Med-PaLM, an adaption of large language models to the medical domain. Topics: * From India to UT Austin to FAIR to Google * Integration of AI into products * Organizing research orgs in large companies * Applications of AI to medicine * Med-PaLM and the limitations of LLMs * Risks and rewards of AI driven products Links: Med-PaLM paper: https://arxiv.org/abs/2212.13138 (https://arxiv.org/abs/2212.13138) Follow Vivek on Twitter here: https://twitter.com/vivnat (https://twitter.com/vivnat) Follow your hosts:John: https://twitter.com/johnvmcdonnell (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbERVM0c3RjBLXzNpa0txY20wNHZSTEp0VFNTQXxBQ3Jtc0ttTVM5VkV3ZmFwUzgxd1prODZ1NWhZbW4yY1FUb3JJdmVxX1JQUFJBVWNLRXNRYUVuQ2haY0xTanlrV0FSNXNRbE8wdjB3YUtkUVlucUFmUERWazJmbGRjX0RCWHllZWQ0Xzh0RnhSdHNHZXJGWTdLTQ&q=https%3A%2F%2Ftwitter.com%2Fjohnvmcdonnell&v=aGoKNj0vcD8)Bryan: https://twitter.com/GilbertGravis (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbDlHdzZNZFkyU3NqSzJkUTBka3Baa3F3RTVuUXxBQ3Jtc0trLVJuWkk3SHBvWVVuOVFCRUdlSXpQc2U4bzlqVlBuR1JwOXNvZWZ1U0V2RnIwcWFrbU00bkZMaE1zcUtWODc1b3VaS0pJZ1ZUa1FsTDJvZ05QbkxKT2RSTWRZTHJ6MXNORmprVTVwUmdTYmdKRFVGRQ&q=https%3A%2F%2Ftwitter.com%2FGilbertGravis&v=aGoKNj0vcD8) Interview Transcript [00:00:00] hi, I'm Bryan and I'm John. And we are hosting the Pioneer Park Podcast where we bring you in-depth conversations with some of the most innovative and forward-thinking creators, technologists, and intellectuals. We're here to share our passion for exploring the cutting edge of creativity and technology. And we're excited to bring you along on the journey. Tune in for thought-provoking conversations with some of the brightest minds of Silicon Valley and beyond. Bryan: Welcome to today's episode. I'm Bryan, and this is John. And today we're joined by Vivek, an AI researcher at Google who has been working on translating AI and adapting AI to usage for clinical medicine. He's the co-author on a recent a paper from Google about Med Palm, which is an adaption of the Palm model from Google to the domain of medicine. We're looking forward to talking to Vivek about all the ways that these models are powerful and useful in select domains like medicine and also their limitations. So we're looking forward to [00:01:00] talking about Spoonerisms, confabulation and hallucination and how all of these words apply for the purposes of AI. Vivek: Vivek Welcome. Hi Bryan. Hi, John. Excited to be here. And yeah, talk all things AI and medicine. Bryan: Cool. Yeah. Welcome. So, Vivek, just to sort of ground your background you did your undergrad in India, then you went to UT Austin, and then you came out to the Bay Area after finishing your master's degree. Is that correct? Vivek: Yeah, that's right. Bryan: Cool. I think we may have overlapped in Austin. I lived there for a number of years. I miss Austin a lot of the time. But I curious to hear about your own sort of migration gradually as you made your way west to California. Vivek: Yeah. I think Austin's a beautiful little city and I think Bryan you wouldn't disagree with me if I say that. I think the school UT Austin adds to the charm as well. And for me it was like coming from India, which is a warm weather place moving straight to Texas and Austin, which was equally warm, was good. And yeah I enjoyed the scenery over there. It was a very welcoming environment I would say for graduate students. And I [00:02:00] was also transitioning my major from electrical and electronics, more hardware after, more computer science than AI. And UT at that time felt like a very good place to be in. Had a number of good professors who were doing some amazing research and natural language processing, computer vision, graphical models and robotics as well. So yeah, I really enjoyed my time over there. Bryan: Awesome. And you found yourself now working at the absolute frontier, I think, of artificial intelligence at Google. Tell us a little bit about how your experience, how did you find your, the application of interest or this domain of medicine? What kind of drew you to it? Vivek: Yeah, it's a funny story because even before deep learning, like when I was doing my undergrad back in 20 20, 20 11, in the final year of my undergrad, we were asked to do like thesis projects or pitch ideas, and the idea that I pitched together with my team was actually an AI doctor. And at that point of time, the planning wasn't a common term, it wasn't invented. So my presentation decks all had, support vector machines and all those kind of things. But I still believed in the potential of the technology because it was very clear that if it did not have tech [00:03:00] and AI scaling of medicine, we are not going to be able to scale world class healthcare to everyone. It was quite obvious to me even back then, and especially coming from a place like India where, the medical facilities aren't the greatest. It's, it's getting there. There's been massive improvements in over the last decade, but reaching, the remote villages and towns is still a huge challenge, I would say. And it felt very natural to me that tech and AI would be the place to be. And so at the back of my mind, I think that was always a place that I wanted to work in. And obviously I had a huge interest in machine learning and AI back then. I remember back in undergrad we didn't have the best of internet connections, so whatever bandwidth I could secure, I would download these courses from the KelTech professor Yaser Abu Mustafa and learn about machine learning. And it wasn't taught in our curriculum back then. So it was all on the sidelines, but that grew, that drew me into the field. And so when I came for my masters, it was, I wanted to take as many machine learning courses as possible. And when I joined the industry fulltime again, I wanted to do machine learning. And I got really fortunate that as soon as I came out of grad [00:04:00] school and went over to Facebook, it was when Facebook AI research was started and I got this incredible opportunity to work. At the intersection of research and product. So I had this nice role where I could take the latest and greatest models from fair and put it into production. I learned a ton over there. So it involved like learning all these machine learning frameworks at that point of time, torch and Cafe. Not easy to use by any means. But it was fun and, getting them into, products with like millions of users. That was incredible learning. And like when you work at that low level, you learn all the details of these models, both at training time, both at influence time, and, learning about optimizing. And so that was, it was lucky for me in the sense that I got this opportunity as someone who was a relative. Nobody just had taken a few courses. I did not have a PhD, but that just set me up back in 20 14, 20 15. And since then yeah, I would say just have been very fortunate to work at the frontier of AI and deep learning since then. Bryan: That's awesome. John: Yeah. And so you have to work at both Facebook and Google. How are the cultures of those two companies [00:05:00] different and how would you describe them both? Vivek: Yeah, it's a great question. I think the answer that I would give with respect to Facebook is also kind of outdated because when I was at Facebook, it was still like, you know, a few thousand employees fair was just getting started and fair was, maybe you could even think about it as like a research lab or a startup within this big company. And My experience over there is, yeah, it was incredible because there are all these stories about like, you know, mark Zuckerberg having the AI team sit right next to him. Those are all two. We used to sit right where like the exact team was there and we would be observers to all the meetings and everything that was going on over there. And he, at the end of the day, sometimes he would just pop over and ask questions. And maybe, and he was also doing some projects at that point of time. These site projects or like projects were, I think back in 15, his project was like an Ironman kind of thing. A speech recognition system or voice command system that you can, could do like house task for him. And so it was kind of, fun that the systems that he was using at that point of time to build this voice control assistant or whatever, was actually the speech recognition systems that we were building at Fair and [00:06:00] Facebook AI at that point of time. And so, yeah, high visibility, I would say. But at the same time I think Facebook is one of those very interesting companies, at least back when I was there, where it did not feel like a big company. It felt like a startup. The culture of a startup . The leadership did an exceptional job in scaling it. And even until 20 17, 20 18 there weren't like, you know, as many processes or bureaucracy in terms of getting things done. Yeah, the goal was always to like, you know, just shift things. If you have an argument don't waste time quantification, rather just build and ship and show me that things work, and then, yeah, that's it. And so I really enjoyed that culture where it was all about shipping. So I would say that was probably the best part about Facebook. If I were to maybe say, Okay. What wasn't great, it was that sometimes you get so much lost into the weeds and details that maybe you don't zoom out and look at the big picture. You maybe micro optimizing for certain metrics and you just keep on moving fast where you're doing okay. Like, consider all the potential issues that might crop up as you're making advancements on certain [00:07:00] technologies or building certain things. And so that has obviously manifested itself in different ways since then. But yeah I think that environment was really awesome for builders. The period that I was there, and especially for AI researchers as well, were really well taken care of, provided with all the resources. I remember there's one particular interaction between the CTO of Facebook and Ross , who's a very famous computer vision researcher, like the builder of object detection, some of the greatest object detection systems. And it was at one of these gatherings Shep came up and he basically said, I would literally sweep the floor for you to, do whatever you want. And so that was the level of privilege or access to resources that you had as AI researchers had at Facebook, at least when I was there. Bryan: Is there, oh, is there a difference in the way that I guess AI you feel has been integrated into the products at Facebook versus at Google? It sounds like perhaps Google and I realize, that you're still there, so we can't just, dive into the weeds, but I'm curious, just at a high level, do you feel that one has been a little bit more strategic or meticulous [00:08:00] about the choices of when to adopt systems, of course, and Facebook maybe shooting a little bit more from the hip. What would you say maybe is a thematic difference in the way that those companies have integrated AI into their products? Vivek: Yeah, it's super interesting. For a long period of time while I was at Facebook, we did. So the FAIR Computer vision team was probably among the best. It still is the best, but there were many other areas where it was feeling like we were playing catch up to Google. So I was at Facebook when the, when TensorFlow was released, and I remember one of the most amazing, machine learning frameworks, hacker just saying that, oh, this is something I wanted to build. But it would've taken me like another year. And if I were to like ask a fairy or a genie to bring me something and put it in front of my house, this is what I would want. And there were like similar reactions when, for example, the transformer paper came out. We were like all shocked, oh my God, how well does this work? And at that point of time, I don't think people realized how important this transformer paper was going to be. But still it was quite obvious that this was. Going to change things. So for a long period of time at Facebook, it almost felt like [00:09:00] we were like catching up to Google. Like all the inventions were mostly happening there except maybe in computer vision. I think that has maybe changed a little bit now. FAIR has it's own amazing research in a few areas. think some of the work on proteins that has happened is really awesome. And also the work on embodied agents and habitat, the environment. I think that's all really cool. With respect to product integration. Yeah, I think one of the cool things was By there was this explicit goal of moving as fast as possible from research to production. So I remember like one of these conversations where I think it was probably Sumit Genal someone who when say, I want to take the model from here and put that into production in two weeks, and actually went out and executed by, enabled that. So when some of the Mascar CNN models came out I think in three to four weeks, it was in one of the internal build demos. So that was like really, really cool. And so Facebook had, at least on the computer vision side, built up this mechanism or like way to like productionize things really, really fast. And the, and I [00:10:00] think Pieto and Cafe both played a huge role. Pieto more recently. But cafe's role should not be cafe's role should not be underestimated. Then. Who there at along Andrew? They were all like incredible and I. I think Facebook was visionary on that front. TensorFlow is great but I think the life cycle from research to production is probably longer than say with Pito, at least the version of PTO that I'm talking about back in 20 17, 20 18. And with the switch to computer vision, Facebook had that incredible advantage where they could like, you know, immediately shift things to the app straight from whatever, like timing here, or Ross or other people at Fairway cooking up . And I think that has slowly caught on in other views as well with speech, with nlp. But that I thought was incredible. That was visionary. John: Yeah. That's really cool. Cause actually a question I was gonna ask, and maybe that partly answers it, but it's a challenge at these big companies when you have a kind of skunkworks team that's like doing very cool research and then you have like a completely different product team that's okay, I'm in charge of like serving up a great newsfeed or something. And then how do you move the insights and innovation from the skunkworks [00:11:00] to that production side team. how did that actually work? I mean, and so obviously like from a technical perspective, it sounds like PI Torch made it easier. There's still, it's still like it's not, it's easier, but it's not free. And then there's also like kind of a communication challenge cuz it's like, oh well, like what should the PMs be building? And if you're in a research team you might not exactly know what the product really needs and kinda vice versa. People on the product side might not realize like the impact they could have with research. I'm curious about how that's been solved at places you've worked. And I'm also kind of curious about maybe your perspective on how it should work. Vivek: Um, I think it depends on the goals of the research organization and the wide area AI organization as well. Right. So, if the charter of the AI organization is like to actually serve the bottom line of the company and be integrated into products as soon as possible Then Yeah, for sure. I think you need to organizationally be aligned with this goal and build up everything like the communication systems, the infrastructure, everything to ensure that you can rapidly deploy and get feedback and improve the models as much as possible. And I wouldn't say this was uniformly happening throughout at Facebook, but the computer vision team was, I think very unique in that [00:12:00] sense because there were very good relationships between the fair researchers and the applied ML teams and as well as downstream customers. And they were all like, putting together in the same direction in the sense that they all wanted to have the latest and greatest advances in know, shipped in the apps as soon as possible, but in maybe some other areas like, you know, new Street ranking. I think it took a few years, for example, to transition over from some of the logistic regression models or even for ads to like deep neural nets. Obviously they were like huge wins, huge lifts and metrics, but it wasn't like two weeks, it was more like two years. And I think that is both a mixture of How the organization is set up, as well as maybe some of these areas you are a little bit more reticent to try good stuff because there's risk associated with it. Whereas some of the computer vision stuff, for example, were like more playful features or features where if you go wrong, it's okay. Right? Whereas I think if you go wrong with your heads that hurts your bottom line. So you probably don't wanna screw that up and see, we really wanna be medicist. So at the end of the day, I think if you really care about getting your innovation to people as soon as possible, then I think at all levels of your organization, you need to be aligned. And one [00:13:00] thing that really helped was I think that leadership was great. I think they finally also re rehabbed so that Research and the applied ML teams were all like reporting to the same V page around. And so I think that also really helped. So yeah, I think you have to be very intentional about your organization if you want to like move fast and deploy. And I think Facebook got that right for a long period of time. Bryan: And on that topic I'm curious, we're talking about an article where the application is medicine and obviously what's at stake when you're giving people medical advice is it's, you know, equivalent if not far more serious than anything having to do with the company's revenue. And what is the kind of ambitious context that maybe caused the research group at Google to pursue the application of medicine? What do you think was the sort of pie in the sky goal? Vivek: Sure. think Google has always been at the forefront of medical ai. Along with say, the transformer paper and 10 flow. One of the papers that I was really, really inspired by was this computer vision paper from like my current teammates which showed that you could detect diabetic retinopathy from fundus images as well as [00:14:00] like eye care specialists. And so that I would say was a very big personal moment for me as well in the sense that, okay, that really it, it showed me that, with ai we can do some amazing things in medicine. And it goes back to the same story where it is very obvious that healthcare systems worldwide have like different sets of challenges. But one of the key solutions to these challenges is tech and AI in particular. So. In developing countries there is just a shortage of like specialists and care providers and probably the best way for us to be able to like scale world plus healthcare to everyone is through ai. And in places like the UK and the US it's more that we do have providers, but their time is occupied in not providing care, but in everything else around it. And they are experiencing levels of burnout never seen before. And again, AI is the solution to help them have a much better experience in providing care. So yeah at Google one of the great things is like the [00:15:00] investment in medical care has stayed consistent or increased over a period of time. Different efforts have been made. And that is really inspiring for me. And I would even go out and say that probably the most important application of AI is medicine. And in the next decade, we are going to have a transformative impact using AI in medicine. John: I mean, one thing I was curious about is, you know, there's that challenge within a company of like, okay, how do you get the research moved in to like production in medicine it's actually a way more complicated, right? I mean, so you have this diverse group of healthcare providers, you have certain companies that are investing deeply in, in medical ai or researchers that are investing in medical ai. And I'm curious like you've kind of got this front row seat. Like, what does that process look like? Like how does the research turn into clinical. Vivek: It's definitely an interdisciplinary process right. I think you can't just have engineers and, machine learning scientists working on this. So if you look at [00:16:00] our team, we have expertise. We have some of the best clinicions in the world who have worked, and it's not just from the US but also UK and Australia and a bunch of other places. We have people who have expertise with respect to regulation, who have worked in FDA or like equal bodies elsewhere. We have like legal folks and everything. And so you need all those perspectives to come in just because of the nature of the field that we are in. And I think it's a mix of what are the most interesting things that you can solve in this place as well as, okay we have this technology where we have this unique advantage or this superpower. And how do. Make best use of that. And so at the intersection, like the magic, and that's why we kind of focus on, okay, find out what are the most interesting problems that you can solve with this technology that we have access to? And that's how we generally end up like picking the problems to solve or whatever projects that we work on, and generally you are looking for the biggest impact that you can make. And so the kind of diseases that you go after, if you look at it, they're like, you know, diabetes or cancer or neurological diseases, [00:17:00] which probably have the highest footprint across the world. So if you make a dent over there, then the quality impact, the quality of Jesus life is that you can improve by that's significant then. So that's how we end up choosing our projects or the work that we do. John: Yeah, well actually, here's another way to put it. Like for example, if you think about. The work that's already been done in AI with medical applications what are some of the big wins so far and like how did those get into clinical practice? Vivek: It's a great question. I don't think I would be wrong if I say that actually the promise of AI and medicine has not really translated into real world applications. There's been tons of research papers. I think there's 150 x increase in the number of research papers here in the US since 2016. Yeah. In the medical AI field in particular. But if you look at, say, the number of clinical trials that's lagging behind. More recently there have been quite a few FBA approvals, especially in radiology for using AI applications. But I would say with respect to the research and the promise and the hype, the [00:18:00] translation hasn't necessarily been there. The ones maybe that are most prominent so far have been in medical imaging. And I think that's probably due to the paradigm of AI and d planning that we've been using so far to build medical AI systems, which are still based on like, supervised learning, acquiring large amounts of data and computer vision at least. Still you know, GPT three came out was probably the most advanced field of AI and medical imaging had this nice cleaned up... I wouldn't, okay, not necessarily cleaned up by natural image standards, but generally you had data numbering in the millions from different hospitals, probably easy to like homogenize and clean up. And so it was very well suited to the supervised learning paradigm. And so that's why you saw a lot of activity and momentum and applications in the medical imaging slash computer vision phase. And so, I would say that's probably the most advanced. We've seen applications in radiology you know, there are different startups doing like breast cancer detection models. There are and lung cancer detection. And then other modalities like the ophthalmology modality that I talked about, like [00:19:00] diabetic retinopathy, a bunch of other eye diseases that you can predict from kind images. Dermatology. I think there's a lot of startups and who are like building these apps that can diagnosis skin conditions from smartphone images. Yeah. Ultrasound is another important modality that's becoming prominent just because of the cost effective nature of the sensor. And so you can do a lot of interesting things with it. For example recently from our team at Google, we showed that you could predict gestational age from ultrasound and you can do it very accurately. And so this is cool because it's a cheap sensor and it's a cheap model that you can put on the edge and give it to community health workers and you can like empower them so they don't need it to have access to an expert Iens yeah. Overall I would say medical imaging has probably been the one field in AI and medical AI that has probably had the most set up advances with re respect to the research that has been done, the number of papers and also the number of products that maybe are going through or have gone through FDA approval. So that is there. I think EHR is another modality where people have been trying to come up with operative insights from your EHR data. Typically in, hospital like ICU settings. [00:20:00] But one of the challenges is if you work at a typical icu, like we have this recording at Google where we just have something which shows like, what does it feel like to be in an ICU setting? And so they're like thousands of buzzes, right? And like every minute you're bombarded with notifications and everything. It's really, really challenging. And so if you have an AI system you don't want it to add to the noise, rather it should give you a very unique insight. And that I think is still challenging. So I would say the applications on that front, like using EHR or to like, predict test or medications or predict some interesting stuff like sepsis monitoring from records or something like that. I think that hasn't been successful or that successful just because of the nature of the problem and also the workflow. So, It's important that you not only consider how good of a model that you can build, right? But it, I think the key aspect is also to consider the workflow that where the model will sit in. And so you can have a very amazing model but if it is inappropriate in the workflow, then it's [00:21:00] not gonna be helpful at all. And I think that's the real challenge. Like, if the research is done without you know, accounting for perspectives of doctors or people who are actually on the ground, then you're gonna miss out on this insight. A lot of research that has been done today has probably missed out on this insight. And that includes things like, for example, selecting the operating point of your model. You want to ensure that you send in the right amount of alerts or notifications or do the right amount of recalls because anything less or more, you're adding more burden to the system rather than actually helping out. Bryan: You know, we think about the application of models and one thing that's a bit of takeaway for me is we often need a fairly risk tolerant or a fault tolerance setting because we need to be able to, you know, ascertain when the models are making mistakes and we need to be able to offer points of intervention and confirmation from professionals who are practicing. I'm curious if we're thinking about the balance of kind of opportunity to improve people's health versus risk of making wrong decisions, we often specifically for medicine have a very conservative threshold [00:22:00] and a conservative approach to this, where we are very, very risk averse and not very opportunity seeking. And that might make sense in an environment like the United States where these, as you mentioned before, the established system functions more or less in that people are able to get healthcare. And that's probably true in much of the industrialized world. But I'm curious if you think that in countries where the medical infrastructure is less established, if there's a benefit to being more opportunity seeking, even if that does potentially raise risks or the risk of making mistakes is sometimes higher. Vivek: Yeah. Great question. It's a hard one as well, right? I am all for more medical ai but you want to be responsible that it's researchers. And so if for example, you built a model only using, data from Western institutions and you're gonna put that in, say a place like Africa or India, it's pretty obvious it's not going to work. And that's I responsible on your part. So if you've done the legwork where you've actually built a model like used, [00:23:00] sourced diverse training data and actually validated in the appropriate settings, and you've seen that it works, then yeah, for sure. We should, I think maybe dial down our risk tolerance a little bit more and be more proactive in terms of deploying these technologies. Yeah, with every opportunity comes responsibility. And there's no free pass. I think you still have to do a good job at validating, but I'm with you. I think there is I would say like there, there's a lot more opportunity maybe beyond the US or places with more established healthcare systems in terms of deploying these systems ahead of time and getting feedback data. And it's probably possible that you might actually see these countries adopt AI faster and make actually, and have a leapfrog in how the care is delivered in these countries. It's the same with for example the financial infrastructure, right? So 10 to 15 years back, I would say China and India were like lagging behind the US and credit cards were dominating in the us but now I feel like US is further behind. I haven't been to China, but have heard stories and in India We don't have credit cards, but it's all digital. And the ease of [00:24:00] doing transactions with micro transactions and micro transactions is order of magnitudes higher than in the us. And so it might, this is an opportunity for these countries as well, I feel like by adopting AI to have a improvement in the healthcare systems. And maybe they go even above what's available in Western countries. And I can totally see that happening. Bryan: I'm realizing that we've gotten this far in track conversation without describing what MedPaLM actually does. So maybe for listeners out there, we should what exactly is MedPalm doing? How do you interact with it? How does a researcher interact with it, given that it's not open to the public? Vivek: So I will start off by giving the motivation for this work. Obviously large language models have been the rich in the wide area community. And medical AI and particular tool data. If you look at a lot of the models that have been developed, those are all like narrow single task supervised systems. But on the other hand, medicine is an inherently humane endeavor where language is at the center facilitating interactions between patients, between clinicians, between researchers. [00:25:00] And generally if you ask like clinicians or patients who interact with medical AI in different settings, one of the chief complaints or concerns would be, oh, I wanted to better understand the model one to more interact with it. But all this model gives me is a prediction with a property and I don't understand why that model is giving me this prediction. Right? So that needs to be solved if you want broader uptake of medical ai and that can be solved through language-based interactions, and that's what language models helps us to do. So that was one of the, I would say the chief motivation of this work along with the fact that obviously there is a school technology and we have access to these models. And if you look at the work in general, we have considered a broad variety of medical question answering tasks. These include like medical exam tasks medical research questions, and also consumer medical question answering systems. And we wanted to benchmark and see, okay, how effective are these models and these different potential end user applications? And so the target user could be a medical student, could be a medical researcher, could be actually a consumer who has a medical information need.[00:26:00] So that's where we started. That was a motivation. And we had access to this model called Palm, which is amazing. It's not open sourced. I don't think it's going to get open sourced. But Yeah, the paper is a way for us to like communicate and get feedback as to what we are doing. And I would say the results that we report with respect to like performance and certain data sets that is not maybe as important as say the evaluation benchmark that we are setting up, or the different axis that we propose to evaluate the answers. And I think this is an iterative process which involves multiple rounds of dialogue between not just AI researchers, but also like clinicians social scientists, ethicists, because I think medicine and even patients and because ultimately at the end of the day, I think you require participation from all these folks if you want to really advance and accelerate the adoption of these technologies and models and medicine. And if you even leave out one community then that's going to come back and bite us out. And so we wanna ensure that, ok, this paper is not meant, you have this fancy model that's more like, you know, we have this model and this, these models are going to come now let's build them out in the right way so that it's applicable to all. John: [00:27:00] I find it even interesting kind of the approach of training a model on medical data as opposed to say, like, you give PaLM access to a bunch of information that it would read and use to answer questions. And something that's kind of interesting there, is kind of like distinction between approaches that involve embedding a lot of information in the model parameters versus having the information be external in some way. And how do you feel like that's gonna end up coming together to make systems that are really useful and robust? Vivek: I think it's always going to be a mix of both. It's this classic system. One was a system two thinking of the debate that goes on, right? I don't think we can. Or it's one versus the other, rather it's mix up Both. Large language models are more of the system one kind but I think over the next few months, over the next year, what you're going to see is more like retrieval style models, which are going to allow you to do more system two style thinking and inference. I think with these models, obviously when you are training on internet corpus internet text, there's obviously gonna be medical content in there in different flavors. Some of them may be accurate, some of them may not be. But the model has seen this. So that's [00:28:00] good because outta the box we do see that these models can answer, but they do understand medical terminology. So if you ask like a model Okay, can you explain this condition? Yeah, it does a decent response but the challenge is really medicine is an evolving field. And so there's always new research being published, a new guidelines being published, and see we want to feed in that context information into the model, and then teach the model how to use that context information or additional information and integrate that with what it already knows which is included in the parameters of the model, and then come up with the appropriate responses. And so, yeah, it's gonna be mixed up with that. I don't think it's one versus the other. John: Yeah, I'm just wondering if there's like a way of thinking about it. Like, is it like kind of like, one way you think about it could be like, oh, like should I think about it like the vocabulary? like, okay, I really need the model to have the right vocabulary and I can't just like teach it vocabulary in context very well. And so that's what I'm getting out of tuning it on the domain or is there more to it? Is there like different types of reasoning that happen in a medical domain? I mean, I know people have had this theory that like, oh, maybe chain of thought comes from code and so like training your model on code is important for that. You know, I'm just kinda curious like [00:29:00] what sorts of things you feel like the model's really getting from the fine tuning that you wouldn't be able to do from say, context? Vivek: So I wanna clarify that the amount of fine tuning that we do with the model over here, the MET model is actually not that big. We're using on the order of a few hundred examples and we are doing prompt tuning. So it's not even the end-to-end model that's fine tuned. It's just these additional soft prompt parameters that we learn and. Our hope was that doing this would help condition the model. The one of the assumptions that we have is a lot of the medical data is already encoded within the parameters of the model. But then at test time, we want to do two things. One is actually point the model to use that information. So this is like looking for a needle in the haystack. And so the model knows about science, it knows about, you know, random stuff on the internet, but you want to condition the model into that, for these set of questions. Use your medical information, user clinical knowledge. Yeah. Right. And so that's one of the things that this helps with. And the second thing is in the medical domain, there is a very unique way of [00:30:00] answering things. There's a very unique way of reasoning about things, and we also want to encode that information as much as possible in these soft, prompt vectors. And I don't think we've. Yeah, it's possible that there's a limit to what you can achieve with these soft pro factors, because at the end of the day, it's still like a few hundred token millions of parameters. But at least the impression that I get is you can get the model to understand the stylist technique of the domain. So if you look at the responses of the model generates, it's not overconfident rather it's more subdued. It clearly says this is what I know. Anything beyond this, you should probably go and seek specialist care. And it also learns to trim down the length of its responses because it knows that anything extra, which it's uncertain about it could be incorrect and that could have downstream consequences. So those are the kind of stylistic natures of the domain that you can also encode. I think that's what is probably happening more, it's not knowledge that's encoded in, it's more like conditioning to work well within the domain. Bryan: I'm curious are there any techniques that you are really excited about [00:31:00] for grounding things in sort of external information? So for instance, teaching these things to basically not rely on their system one thinking, but to kind of know, oh, I do need to go fetch this. I do need to go look this up and verify that. Are there any approaches to that that you are particularly excited about or you think are gonna make progress on these problems? Vivek: Sure. I think there's been a class of models which point to this direction, web GPT, retro and a few others. And being demos from like a few startups, Neva and publicity as well, which are going towards getting, like using search and using that as additional context to answer questions and then also citing and attributing the sources. And so, there are a few different approaches, but I think overall they're kind of all the same, retrieve the right information, feed that into the model, and let that model integrate that information with whatever it's already encoded in the parameters already, right? I think the cool part is it seems to me that teaching this kind of behavior to these LLMs is probably quite data efficient. You don't require a lot of examples. It [00:32:00] seems like even with like maybe a few hundred or a few thousand examples, you can tease the model to learn this generalized behavior. So that seems pretty cool. And so this goes down into tool user, right? And search and retrieval is one of the tools that's in the in the models repertoire. But you can also imagine this being generalized to say any expert in the loop. And that expert could be a human in the loop or it could be another ai or it could be anything else. It could be a calculator, for example. For me, the most exciting part is it feels like this kind of behavior is learnable without a lot of examples and it also generalizes, but I think, yeah, we need some research papers or maybe someone at Google or Open, yeah, I'll publish this. But I feel like that's one of the cool things that's coming up right now. John: When you're choosing your instruction examples, are there domains where you expect this to be deployed that you're disproportionately representing or choosing from? So it's like we don't literally need the model to take the mcat, right? I mean, it's impressive if it's good at the mcat, but we kind of wanna potentially deploy this in a clinical setting. I mean, are you imagining like doctor, is like wanting to double check their [00:33:00] understanding of a certain condition and so they're they're going to potentially ask the model " I was wondering if this medication interacts with that medication or like, I'm doing a diagnosis here, but I have a kind of strange combination of symptoms. Like, what do you think? Like, what specifically are you imagining are our clinical application dialogues? Vivek: Yeah. I think there are fair few intended potential applications over here. Probably the ones that we'll see the earliest are more like educational aids to like researchers and students and trainees. I think we are already seeing evidence of charge being used for like educational purposes. And I've actually learned a few topics just by interacting with it. And you can imagine this happening in the medical domain quite a lot especially with a model that's specialized to that. So I feel like those sort of use cases where, which are non-diagnostic and that means also not safety critical, are going to be the first that we'll see and probably that'll happen in a few months. The second set of application is around like aiding researchers and scientists with respect to information retrieval and citations and similar stuff. I think that that could also be incredibly [00:34:00] powerful. Like if I am writing a paper and if a model could retrieve like 10 more related papers with respect to this paragraph, that'll make my job really easy. Because right now, like I think with every time before a paper deadline, the match scramble is to get your references right and it takes a few hours. I think a model that can do that and summarize it will be amazing. I think that there'll be a game changer for researchers and not just medical researchers, but like all kinds of researchers and scientists. I think the final set of applications you would see are more in clinical settings. And again, I think this is gonna be different. They're gonna be certain applications and clinical workflows, which might maybe involve like extracting information from notes or different documents in clinical settings and summarizing them either to patients or maybe to the doctor centers or people who are working in those settings and like just giving them a very simple intuitive interface. Two, the data under the hood and not like the clunky systems that they have right now. Again, that I feel could happen to your timeline. You could say a lot of different applications. I think we are already seeing a lot of interest from healthcare companies in using these models to do such things. And also there's a lot of [00:35:00] documentation that clinicians generate that they're all like fairly templatized all non-diagnostic. But those can also be automated and have an LM generated summary or like a prescription or like a medication authorization letter or referral letter. So again, those sort of applications is completely non diagnostic totally possible that those things happen within, again, a two year timeframe, if not less. Diagnostic is further down the line. And I think first set of applications we would see would be where there is a, like a human in the loop of clinician in the loop. And it would more be like an information aid system for them where they like have a chat interface to like a database. Similar to Google search, but like a more interactive conversational system where they can ask about interactions of medicine or like an interface to EHR records. And I think those are the kind of applications we would see first. And I think ultimately down the line we'll have more diagnostic systems where it's going to be like, an AI and maybe a clinician or an AI alone coming up with diagnosis based on all the context of information. But that I feel is further down the line. And we have all this research. My hope is this all gets translated very soon, but I feel like that's probably a few years [00:36:00] just cause of the number of challenges that we have solve before we get there. Bryan: Back in 2020. I'm pulling back from your tweet history here. You stated in an opinion, you shared an opinion that some of these applications, the publicly accessible LLMs might be doing more harm than good. And I think they probably, at the time you had in mind the potential to be a source of misinformation, be a source of deep fakes, all that sort of stuff. I'm curious if you're thinking about these things and obviously since then ChatGPT has completely exploded, there's a brand new generation of interest in the applications of these models. Are there Do you still feel like the, that sort of sense of caution? I mean obviously Google has been very cautious about releasing any of its LLMs to the public. There's obviously a very storied history of Microsoft and Facebook having to kind of take models offline because of how quickly they become negative. What do you think about the potential of these things to be open in the public as the APIs? Vivek: It's interesting and I'm glad you pulled that out. I would say I was a bit naive with that I was assuming the [00:37:00] worst and maybe that hasn't necessarily happened. Stable diffusion is a very good example of that a model that's out there openly and people are using it mostly for creative applications. I haven't heard, horror stories or anything about that. And so that does point to a future when maybe these models can be open. And honestly, I would love for these models to be open and democratized, but it's. It would be nice to assume everything is good and everyone has good inventions and just answers because that's not true. And so it's very important that you consider what can go wrong, and different organizations have different levels of risk tolerance. And maybe if you're a startup, you don't worry about that so much because you're not gonna be a legal target. But if you're a big tech company, obviously you have to worry about it a lot. So yeah, I've been very pleasantly surprised by how stable diffusion has gone and how GPT three has also been put to use. But maybe that also has got to do with the fact that these models, yeah, I mean, it's all over your Twitter feed, my Twitter feed, but that's a very small fraction of the people who, interact with the tech or or on the internet, it's still [00:38:00] probably like 0.1% or even less than that. So it's not a mass adoption or a mass feature just yet. Yeah, it's hard for me to know to predict how exactly someone in India or like in some other part of the world who is maybe five years behind what we are how would they use these technologies, and it's very likely that people are all going to put this to like amazing use cases. And I hope that is the case but we need to also at the same time be aware of what can go wrong and build tools and systems to ensure that that happens as little as possible so that we can be more open and democratic about these systems because these are amazing. There are more people we can get them like these into the hands of. That's amazing. Actually just maybe one final point over here. I actually don't know how things are going to evolve because it also feels There are these two competing forces. One is this open AI model, or maybe you can even say, Google's model is where it's the model center sitting in some server or some in some cloud somewhere. And then if you look at Apple, on the other hand they're trying to put like stable diffusion kind of models on the phone. And so those seem to be two competing trends with respect to how these models are going to [00:39:00] evolve. LLMs are a different beast. I think stable effusion, I could, didn't expect that model could be compress and put on the phones so quickly, but that did happen. But LLM I think would be a little bit more tricky to do that. And so it may also be like, which technology wins out, because if you can have like a personal component of element sitting on your phone, then that's really cool and that's another way of democratizing this technology and having access to more people. And that might end up happening ultimately. John: Something that's interesting about Google's deployment strategy is, they've been very public actually about what they're doing. So they have these papers they have not released parameters, which is pretty understandable for most models. With a couple exceptions, like I'm kinda excited about flaunt five, for example. It's cool that they released this parameters. They haven't really released like a product in the way that say OpenAI has. How is Google hoping to have a big impact in the world with the approach they have of not really releasing models either for inference or the parameters? Vivek: I think it's hard to predict how things would [00:40:00] evolve. But if you look at it, open ai, I was also not released any of their model parameters. It's an api. I would say it's very hard to predict. I think what big tech companies in general have is distribution. Yeah. And so perhaps what they're all gonna be looking at is how do we integrate it into our existing sphere of products and, just make them more delightful and more magical for people to use. And that might mean a different strategy for meta or Microsoft or a Google, because they all own different kinds of surfaces, different kinds of products. Cloud is a different question over here. I think people would be hoping that these models stay more centralized and you have a lot more cloud customers, and that's probably a very natural evolution of cloud. But I don't know if that will necessarily play out just looking at how stable, de efficient has evolved. I think what we need to watch out for is how quickly is there a Chat GPT eqp open source? And if that comes out very soon before say A G P T four comes out, then I think the trends are kind of obvious. But that might also, what that might also trigger is maybe [00:41:00] open AI would want to talk even less about its research and be more secretive. And that's not great for all, and that might further slow down the open source application, but, open source is an amazing thing. I mean, this will be people working from, "hey, we're just coming together and creating." I think that it's, so, it's hard to predict how open source dynamics and things really well, but I think that's the one thing that I will watch out for, like, how quickly do we get a chat GPT equivalent. And if that comes out rather soon performing as good as say whatever we have right now, then I think that changes the calculus for everyone. I think. So people are just like, at this point of time, still not sure, and it's mostly a wait and watch game for everyone, not just Google, but for all. Bryan: So you've been able to play with Google's internal tools and you've also obviously have played with ChatGPT. I just wanna know, just from your subjective opinion, which one's cooler, but also after you answer that question, I wanna know what brought you to SPC? How did you find yourself becoming a member of SPC? Vivek: I think chat-GPT is awesome. For me it was one of the most magical experiences that I've had with ai. So I was working on [00:42:00] conversational AI five years back. And one of the projects that I was tasked with at that point of time was to build a system that can help you set an alarm. And what that entailed was me writing out thousands of rules and thousands of different ways in which someone can say, set an alarm. And one night I just said, I don't wanna work in conversation AI anymore because it's not gonna scale. Yeah. And so if at that point of time you had told me that, in five years we're gonna have the kind system, I would've said, you're kidding me. And so for me, Just thinking where we were as a field. And I l P was far behind computer vision at that point of time to where we are right now. I think this is one of the most incredible advances that I've ever seen. I can't really compare with Google Systems, but I can just say that is incredible and I hope to see more and more of these systems. And with respect to SPC, no, it's just an incredible community. I felt. I've always wanted to be a part of s spc. I knew people who are at Open AI or Deep Mind. I got a lot of the people who were very early into deep planning and AI back in 20 16, 20 17. I know that they had, they have SPC sub connections. [00:43:00] So the community have always found it interesting and exciting. And so that was one of the motivations just to meet more interesting people and share knowledge, learn about what people are doing and also be exposed to opportunities. I've had this like pretty incredible opportunity to work with a few non-profits. One of them that come to mind is Rocket Learning in India, which is trying to scale education, primary care education to school children. And through SPC I, I got connected to them and I've been advising them on some of their like using AI in their product stack. And we've been using, trying to use that for grading assignments, but we want to do more personalized content generation and curriculum generation and so on and so forth. And again, just similar to medicine, I think AI is going to have a huge impact on education maybe even sooner. So those sort of opportunities where you have this domain expertise that you have built in, if you can share it more freely, like with people who are trying to do some incredible things in the world. I think that's one of the unique value props that s SPC has, where people are trying to do amazing things and you can, tag along with the journey. And it could [00:44:00] be directly as a co-founder or it could also be like, more indirectly where you're an advisor. And sometimes all you need is maybe just a few hours where you just say, oh, you know, take this model and do this thing. And I've had a few of those interactions as well where people have come back and said, oh, you saved days or months for me. And so those sort of things. And it's the same reverse as well where we call SPC. This is if you're going minus one to zero, right? And you looking for new ideas. And for me, beyond medicine and ai, I am very interested in biotechnology. And I actually think that the next few decades that is the decade of bio and ai, and that a few people have said this before: the amount of biological data that we are generating and how for example, our sequencing technology has progressed, it's progressing faster. For example I think like all levels of the stack single cell data to, clinical data genomics data. So this incredibly rich amount of data that's being generated and biology is messy enough that you can't have like hard rules like math or physics. The perfect description language for that is ai. [00:45:00] And for me, SPC felt like a very natural place to engage and learn more about this field. And I've been fortunate enough to meet like a few people who have product expertise in bio biotech. And so that's been amazing as well. We are putting together a tech bio forum now. It's coming up later this month where we are going to host like a series of talks from researchers, founders, venture capitalists in the biotech space. And the hope is like SPC becomes uh, also the go-to place for people who are interested about biotech as much as say about AI and crypto. And if that happens, I would be really delighted. I think that would make my time at SPC really worthwhile. And hopefully I think there's connections and networks the day I go down the entrepreneurial path, I don't have to look too far to find a co-founder. Bryan: We're glad you're here, Vivek. Thank you so much for being part of our show today, and thank you so much for staying and answering some great questions. We'd like to finish things up with asking you for a recommendation on something you're either reading, watching, or listening to. What is one recommendation you'd give people that you are listening to [00:46:00] you now? Vivek: Actually the book that I'm reading right now is a neuroscience textbook, so maybe I'll stay away from recommending that to I think our last one. Bryan: Yeah, I think the last interview may have also recommended a textbook. Vivek: Principles of Neuro Design. That's maybe it's blur, but Yeah. Yeah, I'm super interested in neuro neuroscience and trying to get inspiration. building more low power AI systems because while I work at a place which promotes large models, I like just looking at how the human body is engineered how low power it is how efficient it is. I think we can do better. So just trying to get more inspiration. So yeah, but I don't know if that's for general audience though. Bryan: That's fine. I think this is an audience of a lot of nerds, so it'll fall in familiar ears. John: Yeah. Sounds cool to me. , Vivek: It's good to know. Maybe we should do a reading group session for this one. I dunno. John: Yeah, you should make a s SPC forum about neuroscience. Vivek: That'd be amazing. This is why SPC is awesome. Yeah. Yeah. Bryan: Well, thanks so much for being part of a part of Pioneer [00:47:00] Park. We're so happy to have spoke with you today. Vivek: Thank you so much. This was great. And as I said the reason for being at SPC is being the opportunity to have these kind of interactions where we can go deep into certain topics or learn more about stuff. And with the peer group and the peer network, I'm just glad that we have SPC and I hope like more smart people decide to come and join us over here. Thanks so much feedback. Thanks John. Thanks Bryan. Take care. See you around. Bye. Bye. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit pioneerpark.substack.com (https://pioneerpark.substack.com?utm_medium=podcast&utm_campaign=CTA_1) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

13. 

    4 Feb 2023

    Alignment, risks, and ethics in AI communities with Sonia Joseph

    Check out our interview with Sonia Joseph, a member of South Park Commons and researcher at Mila, Quebec's preeminent AI research community. Topics: - India's Joan of Arc, Rani of Jhansi [[wiki](https://en.wikipedia.org/wiki/Rani_of... (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbHhLX0V4SzNxRGFRQmFTX01JbFFwT19HODFTZ3xBQ3Jtc0trRGN4a0ltb3RQTkUxcjZMVVJvdXhxMVNzYjJ0RkpNUDl3WDk3S0U2ZzExQUdzbDE1NFFjaU1tb2M3S2l0b2NXanFWMVhUd2paWDlxMmJsdzI4QjFRbjhsUFVndloyQWo0OVlrMzkzTDdBV3FULXlmMA&q=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FRani_of_Jhansi&v=aGoKNj0vcD8))] - Toxic Culture in AI - The Bay Area cultural bubble - Why Montreal is a great place for AI research - Why we need more AI research institutes - How doomerism and ethics come into conflict - The use and abuse of rationality - Neural foundations of ML Links: Mila: https://mila.quebec/en/ (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbGhWaTNWZTlUTFVrWEx1eF91M1FfY2hBZ0tyZ3xBQ3Jtc0tuOVEyeUtlV0NlX1RldnF3RDEyYlBZV3hKQWpmS3VkRmhyUWthMXZzTWtobjlUZnBXb05udVRVcVBwcm91dmRqa3dPR3RGLWtuTEZsV0J5WC1SRzk4N003ZVdoQWpCWUpZRmZTSG1iQlAwc0hHdTBNUQ&q=https%3A%2F%2Fmila.quebec%2Fen%2F&v=aGoKNj0vcD8) Follow Sonia on Twitter here: https://twitter.com/soniajoseph_ (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqa3NCcGpWaXdNdnU4LUl1b0JTemZ4TUZEUE9xd3xBQ3Jtc0tuSTQ1QTM3NzFfczJzYzVhcjdNaHhObXo5blQtcXROcmpYMXFvcUtpZ0Y3TmFpY3dTWFBQSElaV0Q1Wk9hY014Q2VjUm95endkVzVEUVk4ZThsWm9LaThnRjZBaWo3TXp0WFhrVnpPR1l1cC1SOWxlUQ&q=https%3A%2F%2Ftwitter.com%2Fsoniajoseph_&v=aGoKNj0vcD8) Follow your hosts: John: https://twitter.com/johnvmcdonnell (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbERVM0c3RjBLXzNpa0txY20wNHZSTEp0VFNTQXxBQ3Jtc0ttTVM5VkV3ZmFwUzgxd1prODZ1NWhZbW4yY1FUb3JJdmVxX1JQUFJBVWNLRXNRYUVuQ2haY0xTanlrV0FSNXNRbE8wdjB3YUtkUVlucUFmUERWazJmbGRjX0RCWHllZWQ0Xzh0RnhSdHNHZXJGWTdLTQ&q=https%3A%2F%2Ftwitter.com%2Fjohnvmcdonnell&v=aGoKNj0vcD8) Bryan: https://twitter.com/GilbertGravis (https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbDlHdzZNZFkyU3NqSzJkUTBka3Baa3F3RTVuUXxBQ3Jtc0trLVJuWkk3SHBvWVVuOVFCRUdlSXpQc2U4bzlqVlBuR1JwOXNvZWZ1U0V2RnIwcWFrbU00bkZMaE1zcUtWODc1b3VaS0pJZ1ZUa1FsTDJvZ05QbkxKT2RSTWRZTHJ6MXNORmprVTVwUmdTYmdKRFVGRQ&q=https%3A%2F%2Ftwitter.com%2FGilbertGravis&v=aGoKNj0vcD8) And read their work: Interview Transcript Hi, I'm Bryan and I'm John. And we are hosting the Pioneer Park Podcast where we bring you in-depth conversations with some of the most innovative and forward-thinking creators, technologists, and intellectuals. We're here to share our passion for exploring the cutting edge of creativity and technology. And we're excited to bring you along on the journey. Tune in for thought-provoking conversations with some of the brightest minds of Silicon Valley and beyond. John: okay, so today I'm super excited to invite Sonia onto the podcast. Sonia is an AI researcher at Mila Quebec AI Institute and co-founder of Alexandria, a Frontier Tech Publishing house. She's also a member of South Park Commons where she co-chaired a forum on agi, which just wrapped up in December. We're looking forward to the public release of the curriculum later this year. So keep an eye out for that. Sonia, welcome to the. Sonia: Hi John. Thanks so much for having me. [00:01:00] It's a pleasure to be here. Bryan: Yeah, welcome. Sonia: Hi, Bryan. Bryan: Yeah, so I guess for full transparency, John and I were both attendees of this AGI forum. And I was waiting every week's I guess session with baited breath. I thought that the discussions in the forum were super interesting. There was a bunch of really prominent, interesting guests that we had come through. And yeah, it was really interesting some intersection of like practical questions with sci. And a lot of things that are like used to be sci-fi that are getting far more practical than perhaps we ever anticipated. John: All right. So Sonia, I feel like the question that's on everyone's mind is, Who is Rahni of Jansi ? Sonia: Oh my gosh. Yeah. Yeah. So basically like I grew up on a lot of like Indian literature and Indian myth. And she's considered to be India's Jonah Arc. So female leader like has a place in feminist scholarship if you look at any literature. And I [00:02:00] believe she read. Of India against the British. I actually wanna fact check that . John: Yeah, no, that's really cool. Just we love the the recent kind of blog post that you worked on with S and you pointed out how these kind of influences like really enabled you to succeed at your current endeavors. So we're like, just curious about maybe like how your background. Made you who you are. . Sonia: Yeah. Yeah. No, I appreciate that question a lot. So like I, I would say I had a kinda culturally schizophrenic background in some ways where I spent a lot of time. When I was a child in India but then the other half of my life was in Massachusetts. Which was very like a lot of Protestantism and growing up on a lot of like American history. I like I saw things in a calculation of various like cultures and religions and that has like very much impacted like my entry into AI and how I'm conceiving of ai. John: Yeah. Something that we loved about the AGI forum is that you have this [00:03:00] kind of really critical eye towards the culture of the way that AI is practiced and the way that research is going forward. And we can I think you really brought this kind of unique perspective that was super valuable. Bryan: Yeah, I'm curious do you, are there any points at which you think there's like current I guess problems either in the way that research is being done or the kind of I guess the moral framework in which that research is being done? Sonia: It's a really interesting question. I would say the AI world is like very big first of all, so it's like hard to critique the entire thing. But it. Have it, parts of it have some of the problems that physics had in the 1990s or still has in being male dominated or like focused on like certain cultures. And the culture will generate a certain type of research. So your scientific conclusions and the community or culture you're in, you have this like reciprocal relat. For example, in like the 1990s like there's this amazing book called The Trouble with Physics, with Lee [00:04:00] Smolin that goes into sort of like the anthropology of the physics community. And the 1990s, the physics community was deeply obsessed with string theory. If you weren't working on string theory, you just weren't cool at all and you probably weren't gonna get tenure track. Goes into how string theory wasn't empirically proven. It was like mathematically, internally consistent, but it was by no means like a theory of everything. And how the monoculture of physics and like the intellectual conclusion of St string theory would feed off each other in this like that cycle. Lee Smolin basically created his own institute to deal with this problem cuz he got just like very frustrated. I don't think AI is quite so bad. But there are pockets of AI that I do notice. Similar dynamics. And particular the parts of AI that were previously like more influenced by effective altruism and LessWrong in this like the AI safety and alignment camp. I don't think these fields have as bad a problem anymore. There have been recent. Attempts [00:05:00] called the reform attempt that Scott Aaronson had a very great blog post on how AI safety is being . There's an attempt for AI safety, like a legitimate science that's like empirically grounded and has mathematical theory. But I did notice that more classical AI safety definitely had these like 1990s style string theory problems, , both in the science being like not empirically verified, but like dogmatic. And also in the community that was generating it not being fairly healthy. And I guess with the caveat, I'll say have been either adjacent to or in these communities since I was basically like 12. So I have seen like a very long history. And I also don't mean to like unilaterally critique these communities. think they have done a lot of good work and given a lot of contributions to the field both in terms of frameworks talent funding but a am looking at these communities with a critical eye, like as we move forward. Cause it's like, what is, what are coming. Both as a scientific paradigm and as the research community that like generates that paradigm. Bryan: I'm curious. To me there seemed like kind of two issues. I don't know if they're orthogonal but I think like the scientific integrity of a community and the ability for that community to [00:07:00] generate and falsify hypotheses and the culture of that community and whether or not that culture is a healthy culture to be in, whether it's like a nice place to work in and all that sort of stuff. And I guess my hypothesis is like none of us wanna work in a shitty culture and none of us wanna be part of communities where insults or like abusive behavior is tolerated at all. But I think that a lot of scientific communities can be interpreted as quite dogmatic because there's an insistence on a specific sort of intellectual lens that you need to adapt to participate in the discussion. And for me it's it always seems like there's like a balance there. Because for instance, if you wanna be a biologist, you better accept evolution. And like you, you're you have to meet that criteria. And I'm curious, do you think that, for instance in the, is there some sort of almost Intellectual cowtowing or basically a tip of the hat that one needs to do when you're studying artificial intelligence to make it into the room to be taken seriously. Sonia: That's a great question. Yeah and evolution is an interesting example. Cause that's one that has been empiric. [00:08:00] Verified in various places and maybe the exact like structure o of evolution is open to debate. Like we dunno if it's more gradual or happens in leap burst. But example in some AI communities is of accepting that on oncoming AI is gonna be bad. Or like culture or more apocalyptic culture. And this is prevalent in a lot of AI safety communities where in order to. Get your research like taken seriously or to even be viewed as an ethical person. It becomes about character. You have to view AI as inevitable. It's coming fast, and it's more likely than not to be incredibly disastrous. And to be clear, I think ai, like we should be thinking about the safety behind incoming technologies. That's obvious and good. If AI ends the world. That would be terrible. And even if there's a very small percentage that could happen, we should like to make sure it doesn't happen. But I do think that some of these communities like overweight that and make it almost part of the sort of dogma when it's not empirically proven that this is gonna happen. We have no evidence this is going to happen. It's like a priority argument [00:09:00] that's actually like mimicking a lot of. Student stay cults and also like death cults that have been seen throughout history. And it's absolutely fascinating that much to less now than it was before. A lot of possibly AI safety has become like modern alignment. Or practiced in more professional spheres where I think views are a lot more, more nuanced and balanced. But there is still a shadow of Bostrom and Yudkowski and these original thinkers who were influential, e even more influential like 10 to 15 years ago. John: Sonia sometimes when I talk to people who are really into the alignment problem there's a kind of view that like the philosophical argument that is made is just like very strong. And so They just, people just view it as actually just like a very strong argument that this, these systems are very dangerous. If you think about when I think about Holden Karnofsky's pasta, like I, I of imagine okay, I think if the system was that powerful, it seems like it would be dangerous. I don't know exactly how likely I think that exact version of it is to be [00:10:00] created. When you think about those, when you think about the, I guess like the content of that alignment argument? Do you think the content did you just think it's do you think the argument is is strong or do you feel like it's actually overrated? I guess what's your view on that. . Sonia: Yeah. Re remind me of the so my memory of pasta is that there's some like math and AI that starts like, executing on experiments or like using the results of the experiments, like feedback. John: That's right. Yeah. Sonia: Yeah. Yeah. This is fascinating. I love pasta. I, I think it's absolutely fascinating as a thought experiment. My pushback here would be like all of these scenarios strike me as being slow takeoff. Opposed to, someone develops, like an agent, like a single lab develops an agent, and the agent starts like recursively, self-improving and it like takes over the world, which is like often like the classic scenario presented. John: Yeah. Sonia: The reason this doesn't make sense to me is that there are so many like limitations in the physical. for example, just like the speed of molecules in biology. We're gonna be limited by that. The speed of a robot, like [00:11:00] traveling across the country. We're going to be limited by that. Like we there's one, one argument that computers think so fast. They're not going to be able, they're gonna be able to outthink us. I think this is true, but ultimately for the co computer to interface with the physical world, it is going to be dealing with the slowness of the physical world. And that is not something the computer can artificially speed up. There are also various other constraints, like the government has a lot of red tape and bureaucracy. In order to actually run any study you have to go through a certain approval process. Maybe the AI figures out how to bypass that. That's possible. Maybe the AI has a physical army and it doesn't care if that's also possible. But I do think that. , the real world has enough red tape and constraints where we're not gonna wake up one day and see like drones like everywhere. And some like AI has taken over. I think it'll be like slower and more subtle than that. This is also not to say not necessarily to worry, having some sort of superhuman scientist like that gets out of her control sounds objectively bad, but I don't actually think pasta in its current form is an inevitable [00:12:00] outcome. Bryan: Yeah, that's very interesting. I feel like there's like a dynamic with a lot of technologies, and this was true for crypto, the wave of crypto interest is that everyone that was very close to the technology assumed its expansion to take over the world was inevitable. And then everyone far away from it like treated it as utterly irrelevant and it turns out that both are wrong. Like that there's some sort of middle ground of estimating its impact on the world where you can converge on truth. Sonia: Yeah, exactly. And it's so funny cuz like every time I leave San Francisco and go talk to people, not in tech, like I'm in Boston right now, it's just like I, I can feel myself leaving the bubble where people are like, what? What is ChatGPT? Bt like what is, I heard about that on in, in the New York Times. I think , I don't, it's just like a different world out here. John: Yeah. . So maybe backing up a little bit, I'm really curious about Mila, both like the work you're doing there and how you ended up going there and what that community is like. Sonia: Yeah. Yeah. So Mila is a deep learning institute in Montreal. It was founded by Yoshua Bengio, who's one of the fathers [00:13:00] of modern deep learning. And it's an incredible place. Like it's very international. It's the largest like academic institute for machine learning in the world. It has a lot of collaborations with industry. It's known for a lot of researchers who are awesome in reinforcement learning. Ian Goodfellow, who invented GANs, did his PhD there, so it has this like very vibrant history in machine learning. And I went there to work with Blake Richards, who does some incredible work at the intersection of neuroscience and ai. Can we look at the brain fiber AI algorithms and like I felt like Blake's lab had a lot of intellectual independence, which was very attractive in me partially trying to escape some Bay Area monoculture and develop my own intellectual frameworks. I think my intellectual frameworks are very much in development. Otherwise I would be able to stay in the bay and actually interface with that. So Mila's been like a very good environment to, to work on that. Bryan: I'm curious, do you [00:14:00] feel like it's a different bubble? Is it outside the bubble? Is it a loosely connected bubble? Because we, I think we definitely recognize that San Francisco and the Bay Area in particular has a specific ethos and take on the way that AI is affecting things. Obviously this is the heart of the industrial applications or the commercial applications of these things. I'm curious how Mila stays connected but also separate. Sonia: Yeah, so Mila feels like less of a bubble, I think because it's so international. There is a constant influx of students coming in from Europe, from Canada, from India, from the United States that it doesn't feel as closed as an ecosystem. There are also like more stakeholder interests than in the Bay. I think like you have government as a stakeholder, you have industry, you have startups, you have academia itself. So as a result I don't think anyone has to... prescribe to a particular way of being. Plus Mila's culture or Montreal's culture enables this environment in that the culture is very caring and very respectful of individual differences. One interesting thing [00:15:00] that's happening at, that was happening at Mila last year was the introduction of the scaling hypothesis. So the scaling hypothesis, just like the more data compute and parameters you have, intelligent behaviors go to emerge. This is something I think the Bay accepted much sooner and much more readily. For better or worse in, in this case, I think for better. Cause I think the scaling hypothesis is mostly correct, but I noticed at Mila there was just like a lot more resistance to it. And this could be a variety of things. This could be coming from academia. And academia has always been a bit more anti-scaling compared to industry. Just due to compute limitations. . But I also do think that Mila is just less susceptible to any kind of trend due to people just being very exposed or used to very different ways of thinking or more independent in their opinions. John: Yeah, so it can often feel like a lot of research or the kind of vibe of the field right now is just this kind of like scaling hype train. Are there kind of other lines of interest of research that are [00:16:00] you're interested in, that are going on at maybe at Mila or other places that are less affected by that specifically? Sonia: Yeah. One area that I find interesting is some of the interpretability research on toy models or like mechanistic interpretability on, on toy models. So basically you are looking at the structure of the neural net in order to understand what is going to output. So you're trying to reverse engineer its algorithm. And this I think is pretty suited for low compute paradigms cuz you can just have like a four layer, like neural net or some like tiny transformer and just do all sorts of fascinating visualizations to try to find first principles. The hard part here is I do think you need to connect it to the larger models which show, like fundamentally different behaviors. But this is a fun part cuz you can come up with laws that sort of generalized the behavior of these toy models to the behavior of the large models. And you can do a lot of work without as much compute. Bryan: One thing that may be true in the Bay, that the, there's definitely an overrepresentation of these power, of these commercial interests and the startup [00:17:00] culture and the impact of scaling. But one might also argue that is actually the truth. That the reality is that more and more of the innovation in this space and incredible proportion of the kind of overall power represented by these technologies is being monopolized by commercial interests. And that to not acknowledge that, is almost like silly. These aren't, these state actors haven't been very effective and academic institutions are increasingly falling behind. A lot of the best publications are emerging from the private sector. Is there almost like a, perhaps like a blinders on in terms of not acknowledging that. Sonia: Yeah. Yeah. I think that's mostly right. The one caveat is I think academia is still useful for this sort of theory and doing ethics and policy work where that is in conflict of interest with industries in cor in corporations as a safe haven for that. But returning to your question, I think in academia there is, Amongst some parties, maybe some denial about scaling laws [00:18:00] because it might make your research feel less important. It might make it feel like you've been working on an inductive bias for five years or 10 years, which is a very long time And there might be a sort of like psychological, like aversion or denial of it to protect like your research. Plus I do think there's less engineering talent. All these industrial labs are working side by side with amazing full stack engineers, and in a lot of cases, setting up these systems requires that talent and that can feel less accessible if most of your skill is like mathematics or doing stuff in a Jupyter Notebook. John: Yeah. I've heard you ask the question no, I was trying, actually trying to see if you'd written about it, but I'm not sure you have, but about what would it take to create a new AI research group? Is that something you're exploring? Sonia: Yeah. Yeah. That question is like so fascinating to me. Like I, I'm very fascinated by the origin stories of Deep Mind and OpenAI and want to spiritually do something similar, but think that it's going to actually look quite different because when Deep Mind and OpenAI were [00:19:00] founded AI was like less hot, or it was like hot a, in a different way and it was considered more sci-fi and crazy. So there was a lot more alpha in doing something like this. Like the founders of both Deep Mind and OpenAI were called crazy multiple times by both their colleagues and by professors. So if I were to. Say, all right, we're starting another deep mind. We're starting another like industrial AI research lab. Let's go. Of course this is like a very ambitious and like difficult thing to do, but it's now considered to be like very feasible, which makes me wonder if there's like less, actually, like less alpha and doing something like this , so right now, like starting privatized science or like another privatized science. it's this very fascinating high-dimensional problem of what is your funding model? What type of person are you gonna hire? Like how do you balance attention between research and development and product? How do you, how quickly do you spin off into applications? , how do you ensure like the right feedback loops so that your [00:20:00] research is going in directions that are both like scientifically useful and eventually commercially useful? Yeah. Do you go straight to consumer? Do you go to businesses? Do you go to governments? How do you ensure that you're going to have enough funding? When we have like language models that are like in the billions of dollars, which I think is not crazy to happen. So these are all questions that we were actually exploring in, in, in the AGI forum that I'm still thinking quite strongly about. John: Yeah. Is there a gap that you see where it's oh we've got open ai, we've got Deep Mind, like. Where those guys are missing something or like there, there's a need for an additional house that's gonna do stuff differently. Sonia: Yeah. Yeah. I definitely think there could be like 10 and fif 10 to 15 more companies. Like I think the space under-saturated actually. But when you're in the Bay Area, , it can feel saturated. I I think even if one were to try to create another philanthropic the research would inevitably take a different direction just because space of possible research is so big. So it could be another like privatized [00:21:00] science bet or some sort of b corp like anthropic. But there are also like other niches that haven't been filled. For example, some sort of group that works closely with the government. Like Microsoft for example, I do think has people doing this. But I'm thinking about something more like Palantir or Andrew. So that's a niche that's not yet been filled in terms of doing a lot of like foundation model based stuff. And the second thing that comes to mind is, Something that's as independent as possible from capitalism. So some sort of un for agi. Not literally the un but that's like analog where you're working primarily with governments and you're primarily getting government funded. And I think this would also be incredibly useful in like setting international standards for doing this kind of research. Bryan: I'm curious, we've talked a little bit about maybe not being as much of a doomsday doomsdayist regarding an uncontrolled agi like taking over the universe and making us all into paperclips as the famous example is. [00:22:00] But what do you see as the kind of realistic doomsday scenarios or like the bad outcomes that could result from use by state actors? What are the, what's the kind of geopolitical perspective that you have on the use of these technologies going forward. Sonia: Yeah. Yeah. So I am way more afraid of people right now than I am of ai. Like I, I think powerful high leverage AI in the hands of some bad actors is more scary than potentially the AI itself being a malevolent agent. For example one country commanding like an army of like drones and robots that are all have like very advanced, like facial recognition technologies that can invade villages or invade towns or invade cities. This is scary. Bryan: The current reality in Ukraine, basically. Sonia: Yeah. . Yeah. Which is already quite bad. Like scary AI outcomes might look like everything that's scary we have now just augmented. Where that or situation in Ukraine is way more right now it's quite dire. It's way more dire and it's way more of a [00:23:00] global threat because it's AI augmented. And then of course there're like less like sci-fi or like less Hollywood friendly scenarios where all the AI we have is just loaded with weird biases. It's like a bunch of like very biased, like very Racist or sexist humans, like everywhere, just making weird decisions that are full of all these, like structural , biases that we can't quite understand. And ... Bryan: like nazi bots everywhere, basically. Sonia: Yeah. Yeah. Like Nazi bots everywhere. Like in your tax software, like in all these like predictive policing algorithms in your self-driving cars, just like systemically making decisions, and this is already here in, in terms of like certain groups of humans. Which is what I mean by again, like feel like evil is already in the world and evil AI will just look like an accentuation of that evil. John: So actually I, I did wanna ask you about your relationship with Southpark Commons. So what, what caused you to join Southpark Commons and what's your experience been like? Sonia: Yeah, so I'm largely coming from like a research and [00:24:00] academic background, but in like really internalizing the founding story of Deep Mind and in seeing sort of academia dying as Bryan you were talking about or at least dying in certain ways. I very much wanted to master startups and entrepreneurship. So South Park Commons was actually an ideal place to do it because the, it's so exploratory here. It's it is like a Montessori school, in some ways. That it was an ideal place to talk to a lot of founders and see what is a tick to start a company. What sort of mindset and skills do you need? So that was like one. . And the other thing that I really liked about South Park Commons and has been incredibly useful is the open-mindedness here. In the Bay Area. We've been talking about this monoculture, and I don't think anyone's completely immune from say the generative AI hype cycle like that I think has hit almost everyone I know who is in tech. But at SPC I have not experienced a very dogmatic [00:25:00] way of thinking of ai. In fact, like the sort of modus operandi of thinking about AI has been very builder for like builder friendly, like open-minded. And this has made SPC a great place to try start developing frameworks to think about AGI I that are not biased by an a priori doomsday focus. John: Yeah, do you actually feel like, maybe related to that, like there's this kind of like deccelerationist idea coming out of the doomer world. Do you think that might be actually hurting us in terms of our ability to handle the actual challenges that are arising around AGI stuff or AI stuff? Sonia: Yeah, the deccelerationist of stuff is really fascinating. Like I remember at NeurIPS there were a bunch of researchers wearing "Don't build AGI" shirts, . And it's interesting, I'm not sure what it looks like in practice. Is this like a social and like artistic movement or are there actually ... being made that are decelerating ai? I'm [00:26:00] not sure. Like the way I like to think about it is, Thinking very carefully, how are we applying this AI now and how can we be applying it in the most effective and maximally good way. Which is both a technical problem and a a social problem. Are we applying it to. Reduce poverty in India? Or are we giving it to like farmers in India to test the soil so they can plant their crops with more intentionality as to weather patterns? Like that's like a very different use than say, like using it for social media. So I'm not sure. I would say don't build agi, but think extremely carefully as to every single use case that you ... aGI two would be my version of. Bryan: In recent interviews, Sam Altman has talked a lot about how as the cost of most forms of intellectual work decline --and I think that is a realistic hypothesis that like document generation, code generation, like a ton of white collar sector jobs and fairly [00:27:00] expensive skills are gonna become a lot cheaper, and that those are gonna be filled in by increasingly sophisticated models. Now, whether or not we call that AGI or just ai, I, think it's an interesting debate, but the, for me, there seems to be a bit of naivete. On what the future looks like in this world of abundance because the the economic sort of biases seem to be that the owners of these systems which are these large commercial entities that we've been talking about, are the ones that get to absorb all that value and that the entire middle. And we're talking, career transitions that we probably have tr have trouble thinking about. Just massive career transitions for people. And it, there's this constant cycle of destruct creation and destruction with jobs. But I'm curious, what are your own views on how society begins to change as a function of these tools getting more sophisticated? Sonia: Yeah, it's fascinating. One thing is I do think [00:28:00] automation of jobs is largely overplayed. This is seen historically and I think this is the case again, in that jobs will just become different. Like right now, prompt engineering is almost like a job or a thing that some people are better at than others. And when prompt engineering becomes less relevant, maybe these models become better. I do think it'll be something else, like maybe, I don't know, navigating like the nightmare ecosystem that's about to develop around, language models or or setting up like generative AI workflows for people. I do wonder if using these tools is going to be harder than we think. For example, I was trying to use ChatGPT to help me write an essay and found it was pretty useful in helping me reword awkward sentences, but it was like pretty bad at actually getting an essay that flowed in a unique way. And sure, maybe we need more data and GPT4 will solve this problem. But I do think that, that these are just, these are so far tools that are automating away some jobs, but not necessarily making things as easy as we would think [00:29:00] and making a bunch of new jobs arise. But say I'm to take like the strong version of what you're saying, which is yeah, jobs are actually, are getting automated away and this is actually like quite bad and these corporations are owning these algorithms. And we're seeing like massive wealth disparity. This strikes me as being at government level of policy in terms of having some sort of like universal basic income. If we're gonna have a post scarcity world that. Pretty necessary or pretty severely taxing these organizations or treating like aspects of AI as a public good or as a public utility. Bryan: Yeah, one part of me fears that we're headed towards Elysium, I don't know if y'all have seen that movie . Sonia: Yeah. So that's like a terrifying AI future . I, I fear. , I fear that as well. And also very much don't want that. John: I guess actually kinda underlying these conversations are the questions about what systems of ethics we need to be applying. So obviously there's been a lot of impact from the EA movement. Alright. Now we've had some retrenchment on some of that due to recent [00:30:00] the FTX meltdown and Sam Bankman-Fried's association with it. What's your view on maybe both like effective altruism and also like I, what you might what alternatives you think are interesting as well? Sonia: Yeah, I love that question. So I do think we need to be looking at every ethical system that has , that had any sort of prominence, like across cultures. So not just ethical systems from Western cultures or Europe, but also ethical systems from Aboriginal tribes or other countries China, India, Japan. I am coming to the conclusion that there is this like meta ethical system that people have or develop in which you deploy almost like a different ethical framework in different contexts. And it's actually like quite complex. But there is like this almost like meta ethical algorithm that's like quite sophisticated and it doesn't look like something as this as clean and decomposition as like deontology, like consequentialism, like one or the other at certain times or different ratios or it, it's not quite as simplistic as that.[00:31:00] And I do care a ton about something that I think is very underemphasized currently, which is the sort of like personal, ethical, and like moral systems of the researchers, the scientists, the engineers, the operators, the people in this field making high leverage decisions. And I think this is controversial because in America there's often this like strong separation between like your work life and like your personal life and like what you do in your personal life like shouldn't affect your work life. But I think when it comes to anyone in a position of leadership or high leverage, I do think that ethics of both have to be like relatively watertight or as good as possible. And ethical and moral education for these leaders is like pretty paramount and this education could look like studying like these ethical systems and moral systems across the world. It could also look like, just like traveling a lot and like talking to a variety of very different people different ages, like from different countries, like from different like cultures and subcultures. [00:32:00] To just get a better sense of what the world is like. Regarding ea there is just a very strong emphasis on rationality, on utilitarianism, to a degree that I don't think is functional. I think a little bit of rationality and a little bit of utilitarianism is great, but these cultures have tended to overdo it. The most sophisticated thinkers in these communities don't make these mistakes, but I think the average person in the community and the community as a whole does make this mistake. And rationality is especially interesting because I think in India, which is the other culture I'm familiar with, I don't see this like strong clean binary between rationality and emotion. It doesn't make sense. It's like taking a neural net and like arbitrarily saying the logits are rationality and like the inner layers are like intuition, like something like that. They're more viewed as like parts of a more cohesive system. So to like almost fetishize rationality over intuition or emotion. It doesn't make sense from, in like a more structural way. The [00:33:00] other thing is that there is this amazing book called Eurocentricism by Samir Amin, who is like this Egyptian scholar who goes into sort of the history of metaphysics and metaphysical cults. And we'll notice that there have always been cults of reason like rationality based cults like this is not some recent phenomenon. This is a phenomenon that's been documented for the past 2000 years, and the scholar makes the hypothesis that he, he just notices that all of these cults of reason and rationality will eventually dissolve into like vision and revelation. And I don't I don't stand strongly by this statement. I haven't seen enough case studies but it's just like an interesting, like thought experiment where what if your systems of reason are not enough to represent the world. What if in some ways emotion and intuition are actually more predictive or more useful? And our very technocratic society has largely dismissed like huge swaths of [00:34:00] religion. When I think this is like a huge mistake and religion is functional and useful in a lot of ways. My answer to like what moral and ethical systems are like to see in ai... I I think we need to study everything. And not just restrict ourselves to like rationality or western dominated thought. Bryan: To maybe make one part of that more concrete. You mentioned at the beginning of this that in within our own behavior there's sort of an adoption of a specific kind of ethical framework and then we transition between those. Can you provide an a concrete example of like where you think that, get where that happens? Sonia: Yeah. Let me think. Say there is a research community that is working on some very high leverage technology and there's this tension in the community about sort of personal ethics and utilitarian ethics. So here the utilitarian goal is we're going to develop vaccines against viruses. But this is very dangerous because we might accidentally develop a vaccine that doesn't work. Or we might have some sort of security [00:35:00] hazard and the virus might escape. It is our foremost goal to develop a safe vaccine and do this as quickly as possible else, or else millions of people are gonna die. And we will do whatever it takes. We'll sacrifice our sleep schedule, we'll sacrifice our personal lives. We'll sacrifice our quality of life to ensure this will happen. Cuz we believe it's just so important. And in many ways, this is incredibly noble. Sacrificing these like more like local resources for the sake of the goal of saving millions of. In this very like visceral kind of way. However, say some researchers start cutting cutting corners. Some of the researchers, especially the high leverage ones and the ones managing other researchers start screaming at their employees for not working more start controlling huge swaths of their employees' lives. Start sort of manipulating their employees or even taking advantage of their positions of power to psychologically abuse their employees. [00:36:00] So here you see like this tension between utilitarianism and virtue ethics. Sorry, not virtue deontology is what one could say. John: Maybe both. Sonia: What John: maybe virtue ethics is too, right? Maybe both. Sonia: Yeah. Both. And you wonder is it worth dealing with what might be called like local problems or like problems on a shorter time horizon of this like unhealthy community? Or is, or do you say it's worth it? Like, all of this is justified the ends justify the means. And I think there are different contexts where your answer might change. If tomorrow, like literally tomorrow, we were all going to die from a virus and we have very high confidence in this being the case. Then I would say focusing on, like emotionally fixing this community and like getting the right people therapy or kicking out bad actors. There just isn't time to do this in a way that's like effective and humane. But in a different scenario if the virus were coming like two years from now and there is time then maybe all of this is justified . [00:37:00] There's also another view where, abusing people close to you is like never ever, okay. Not even for the name of a utilitarian goal. So here it's like a human rights violation, as treated as unilaterally bad, human rights are sacred. You cannot ends, never justify the means. And I think one needs a maximal amount of context to make this call. I vere on the side of human rights are sacred. Like they cannot be violated for utilitarianism, but know that there might be scenarios where things get increasingly gray. And that very much is a judgment call. The reason I don't like the trolley problem or just I don't like trolley problems in general is because they're all very contextless. What is this trolley like, why are people like tied onto the trolley track? Like who invented this? Bryan: Who put them there? Sonia: Yeah, exactly. Exactly. It's like the context lessness of the trolley problem. And it's making it like overly rational in, in almost making it in this idealized like gasless, physics style chamber is actually like, not that like [00:38:00] realistic and not great, like moral training. Yeah. John: What I love about your example with the way that you're treating people, if there's like a virus that's gonna kill everyone tomorrow or something, is you can really see how that directly applies to fast take off doomerism. Because if you really strongly believe in fast take off doomerism or convince other people of fast take off doomerism then that will like tilt, if people are very utilitarian, then it will tilt people towards excusing a lot of like short-term behaviors on the basis of that, that otherwise would never be excused essentially. Sonia: Exactly. Bryan: It reminds me a lot of revolutionary thinking in general of how like it is often the case that the most egregious violations of human dignity occur by people who are very persuaded in some sort of doomerist or doomsday sort of theology. Be that. Like socialism or re reli religious a religious movement, like these sort of cultish things emerge around a really strong thesis, [00:39:00] which basically serves as some excuse or justification to commit anything anything is permitted under that strong thesis. Sonia: Yeah, exactly. The ideology takes precedent over individual humanity. Bryan: One point, I'm just curious to throw this out. There is, this is relates to a quote, I think it's from Mark Andreessen, and I never quote it correctly, but it's like the magic power of capitalism is to absorb all criticism about itself and then sell you a t-shirt. And I feel the same way about like critiques of rationality, like rationalism, perhaps as a sort of like pseudo spiritual movement can be critiqued and perhaps set aside, but to explore your argument about your own moral framework there. You attempted to structure that in a kind of rational framework of thinking about a time bound and like which which moral framework might be more useful under which time bound? And I would argue that's an application of rationality attempting to solve this ethical question. Sonia: It could be it could be an application of rationality. It, or [00:40:00] maybe it was an intuition that I later formalized into words in order to communicate it. I... would love to see, I guess what is your definition of rationality? Is it like using scientific language? Is it like attempting to use logic? Is it I'm going to try to event invent like words. I, is it like how you're directly perceiving the world? Is it... what do you mean? I guess . Bryan: That's a good question. I guess I would somehow, I think I would define it by, Some attempt to transparently derive one's conclusions through causal analysis, including one's own kind of like predispositions and and thoughts. So for example, I think the utilitarianism is often critiqued as really being not very descriptive of how humans behave and missing something about the locality of consciousness and the locality of emotions. And I believe that locality should be accounted for. That's a fair [00:41:00] criticism. And I think it's actually quite reasonable to care about the health of one's community versus the caring about an anonymous community that one never interacts with. I think that is a rational concern and actually can be driven and explained by not just purely like emotional things, or in fact, the emotions behind it can be contextualized by, "oh yeah emotions are part of the human experience, so we should try to rationally account for our emotional disposition towards things." Sonia: Yeah that, that makes a lot of sense. Like some sort of like integrated view. I think you can reframe a lot of like emotional things in rational language and you can reframe a lot of like religious thinking in rational language as well. And this is also actually my preferred way of being like, as, as much as I'm critiquing rationality I, rationality is great. I love rationality. And do think that the mapping and do prefer like the mapping of emotional stuff to rationality opposed to the [00:42:00] other way around. The other way around doesn't necessarily make sense. Bryan: Sounds like There's a shared critique of people who use this, that dress themselves up in this in order to justify being a******s. Sonia: Yeah, for sure. And also the broader theme of just like humans in general, taking some sort of ideology and turning, converting the ideology into some sort of like power structure and using the ideology to justify like personal gain. For example, like the Catholic church might be like the egregious example of that, but I think that dynamic is seen like throughout history and cultures. John: There's some other parts of your background that I I was really curious about that I was hoping we could touch on too. One thing is you actually have some degree of expertise in neuroscience, right? Sonia: Yes. Yeah. So I was originally actually a neuroscience major and worked at Genia research campus doing neuro stuff. John: How have recent advance has an AI changed your viewpoint about neuroscience? Sonia: It's a fascinating question. I became disillusioned by neuro for a while. , [00:43:00] like a lot of neuroscientists. The brain is just so noisy and complex. And toy models weren't generalizing. When all the scaling laws stuff came out every, everything clicked cuz it's like, of course, complex systems are going to show these emergent properties. And went very deeply into studying AI as being basically almost like studying the brain in a lot of ways, but with systems that were very clean. And in neuro there was been, there was a lot of excitement around using neural nets as these kind of artificial laboratories to make conclusions about the brain. And I think this is still feasible, but it's not as easy as it sounds because I've tried it. And I think I still see neuro as being very relevant to AI going forward in that a lot of the interpretability stuff we did on the brain on fMRIs are useful for interpreting toy models and also models at scale. [00:44:00] Interpreting models at scale feels a lot like neuroscience on artificial brains. And a lot of the same principles will apply. But given my frustration with neuro, I'm entering all of this interpretability stuff with a lot of weariness. Where I wonder if our ability to interpret complex systems will hit some sort of asymptote at a point and interpretability may not be as important as we think it is. And what might be more important is actually aligning these systems in a more engineering like capacity. John: Yeah. Yeah. And I dunno if you know but I actually have a PhD in cognitive psychology and I had similar frustration. I I was even I was definitely like, I, in some ways I feel like I was like way too early. Cuz I w like I was doing mine. I entered in 2008 and it's, People were like over neural nets. Like Connectionism was like not considered a useful model, but then people were just oh, we should just go even higher levels of Mars abstraction and do like Bayesian models and so my my thesis is all about Bayesian models, but it's just like very abstracted from what's actually happening. Bryan: I'm curious to dive a little bit deeper there. My understanding is that there's [00:45:00] obviously this sort of branch of AI exploration that I think Bostrom calls whole brain emulation. And there's been some very simple projects of emulating. The neuro neurological systems of very simple organisms do. Is that a branch of research that just feels very bounded and constrained by the complexity of biology? Is that a branch of research that you think will have some things unearthed from it? Sonia: It's a fascinating branch bridge of research. I think they're definitely gonna learn something, but it's also like a different goal than building artificial systems. Emulating biological systems or understanding the brain almost it takes on these like tones of like naturalism where it's like, what is the natural world like as it is now, and how do we form a very high resolution model of that? And that's like a very scientific and worthwhile endeavor, but it feels like a different goal than what is intelligence in the abstract and how do we build intelligence systems that benefit us. And it's not clear to the extent that you need to look at the brain [00:46:00] .To some extent --yes, like deep boarding originally arose from looking at the brain, so I think there's still a lot of stuff there, but it's not totally clear how biological one needs to go to carry that back. John: One thing I find kinda interesting is just that like back prop seems to not be the brain's algorithm, but it's seems unclear whether we know what the brain's even basic algorithm is, which is, it seems like it would be useful to know . . Sonia: Yeah. That's so fascinating. And that's something that my lab, like Blake Richards has done like really interesting work on if you have like Hebbian learning and like back prop or like various other algorithms, how effective is Hebbian learning, if you like, scale it. Like what is like the scaling law of each learning rule and the scaling of Hebbian learning doesn't appear to be like very good. So it's like, what is going on? I would love to know the learning algorithm as well. . Like right now it seems like propagation is like abstracting over various levels of learning from like individual learning to cultural learning to like civilization scale learning [00:47:00] and back prop is like the right like metaphor for all of that happening at once. Bryan: I heard an interesting thing just to cite an example that did draw a parallel between, a discovery in the computer science setting or the artificial intelligence setting that felt like it had a corollary in biology. And that was time difference learning. And this recently came up with an interview with between Ezra Klein and Brian Christian in which Brian Christian was taking was exploring a parallel that basically the ability of our brain to be gratified from being pleasantly surprised was just very similar to the way that a system like AlphaGo learns from its forecasts being off dramatically. And I thought that was just like fascinating. It's oh wow. Have we threw a separate means, discovered something that has a biological corollary? Sonia: Yeah, that's super fascinating. Like dopamine reward circuits and TD learning is very cool. . Super cool. John: Okay, so I think we're starting to come up on time. I thought maybe a good final question would be what advice do you have for smart [00:48:00] young people who are curious about intelligence? Sonia: Ooh it's. broad question. Follow your instincts. Like the field of intelligence right now is somewhat constrained in my opinion. And historically has always been constrained. In terms of defining what intelligence is and the various ways in which we could study it. Intelligence is one of those words that historically was like monopolized by like particular types of intelligence, like being super good at chess. Researchers just made a bunch of algorithms that were very good at chess. But types of intelligence that I would love to see are robots that are very good at dancing or o other forms of intelligence as well. So whatever your intuition is just follow it. It'll lead to interesting places. The second thing that's might be more practical or actionable, is to just read a ton of textbooks. Like you can get textbooks for free online and if you don't necessarily have access to like classrooms, just like download them and just read them cover to cover. And that's how I learned Deep [00:49:00] Learning. It wasn't through school, it was through just reading textbooks and autodidacting, and the knowledge is more accessible than one might think. And you don't have to be super intelligent to study intelligence. John: What are some of your favorite textbooks? Sonia: Deep learning. It's Aaron Courville, Yoshua Bengio and Ian Goodfellow is like a classic. I've read it, like four or five times. Deep learning... I think it's one of those fields where the first time you read a textbook, like a lot of it might, fly above you, but if you just keep like staring at the same equations in a different context for many years, eventually it all starts clicking. Bryan: Thank you so much for being our first guest on the podcast. It's awesome to talk with you. Sonia: Thanks so much guys. That was really fun. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit pioneerpark.substack.com (https://pioneerpark.substack.com?utm_medium=podcast&utm_campaign=CTA_1) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4) Get full access to Ends and Means at gilbertgravis.substack.com/subscribe (https://gilbertgravis.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)

Reach and audience

Public platform figures. Ratings count people who left a rating, not total listeners.

Apple Podcasts (US)

5.0 / 5

1 ratings

Contact Pioneer Park Podcast

Guest appearances

Books guests

Based on episode analysis; this does not confirm that the show is currently accepting guests.