Learn system design wherever you are. Each episode builds from first principles to the decisions engineers face in interviews and real systems, using concrete examples, trade-offs, recall checks, and one-minute explanation exercises. Season 1 covers networking, APIs, data modeling, indexes, caching, sharding, consistent hashing, consistency, and architectural estimates in 16 episodes. AI narration using Liam at a relaxed pace.
S2 E26 — Design an AI Evaluation Platform: Reproducibility, Grading, and Release Gates
An evaluation score improves even though the model has not changed. Daniel and Maya design a platform that can trace the difference to data, graders, configuration, infrastructure, or actual model behavior.
Frozen inputs, task attempts, grading versions, failure classification, and aggregation form a reproducible run. Infrastructure errors and retries must not distort model comparisons. The interview ends with uncertainty and release gates, asking what evidence a higher score really provides for a deployment decision.
Chapters
00:00:00 Welcome and an unexplained score gain
00:02:26 Frozen datasets, models, and run inputs
00:05:29 Task execution, grading, and aggregation
00:08:46 Grader changes and comparison validity
00:12:32 Worker failures, retries, and fairness
00:16:33 Uncertainty and reproducible comparisons
00:20:15 Release gates and evaluation operations
00:31:14 Final score interpretation and recap
Daniel is the interviewer, voiced by Liam. Maya is the engineer, voiced by Bella. AI narration at a calm pace, with three thinking pauses.
Opening music: Stealth Groover by Kevin MacLeod (incompetech.com). Licensed under Creative Commons Attribution 4.0: https://creativecommons.org/licenses/by/4.0/ . Edited excerpt with EQ, volume automation, a smooth fade, and narration.
22 Oct 2026
S2 E25 — Design an AI Research Assistant: Sources, Budgets, and Checkpoints
Three sources tell two different stories and the research deadline is approaching. Daniel and Maya design an assistant that can choose what to investigate, preserve its evidence, and stop within time and spending limits.
A working lookup expands into parallel research, conflicting sources, checkpoints, and recovery. Tool results and pages remain evidence rather than authorities. Evaluation asks whether the answer is supported and useful, including when uncertainty or insufficient evidence is the conclusion the assistant should report.
Chapters
00:00:00 Welcome and conflicting research sources
00:02:30 Answer requirements and evidence records
00:05:47 A complete research lookup
00:09:29 Conflicting sources and uncertainty
00:13:24 Parallel research and bounded budgets
00:17:15 Checkpoints, tools, and untrusted content
00:21:29 Evaluating useful, supported research
00:30:22 Final answer and evidence trail
Daniel is the interviewer, voiced by Liam. Maya is the engineer, voiced by Bella. AI narration at a calm pace, with three thinking pauses.
Opening music: Stealth Groover by Kevin MacLeod (incompetech.com). Licensed under Creative Commons Attribution 4.0: https://creativecommons.org/licenses/by/4.0/ . Edited excerpt with EQ, volume automation, a smooth fade, and narration.
22 Oct 2026
S2 E24 — Design Enterprise RAG: Retrieval, Permissions, and Citations
An employee may read a document when it is retrieved, but loses access before the generated answer is released. Daniel and Maya design enterprise RAG around that permission race rather than relevance alone.
Document versions, ingestion, retrieval, evidence selection, and answer generation form the baseline. Deletion and indexing delays test the copies between source and response. Citations and evaluation ask whether the final answer has adequate evidence and whether the user is still authorized to receive it.
Chapters
00:00:00 Welcome and a wrongly disclosed answer
00:02:24 Source records, versions, and permissions
00:05:41 Ingestion, retrieval, and generation
00:09:12 Revocation races and release-time checks
00:13:14 Relevance, evidence, and citations
00:17:08 Index failures, deletion, and freshness
00:20:58 Evaluating retrieval and answer evidence
00:30:50 Final authorized answer flow and recap
Daniel is the interviewer, voiced by Liam. Maya is the engineer, voiced by Bella. AI narration at a calm pace, with three thinking pauses.
Opening music: Stealth Groover by Kevin MacLeod (incompetech.com). Licensed under Creative Commons Attribution 4.0: https://creativecommons.org/licenses/by/4.0/ . Edited excerpt with EQ, volume automation, a smooth fade, and narration.
Contact System Design on the Road
Guest appearances
Books guests
Based on episode analysis; this does not confirm that the show is currently accepting guests.
Host of System Design on the Road?
Claim your podcast to manage its listing and keep your show details accurate.
Pod Engine is an independent podcast discovery and analytics service and is not affiliated with or endorsed by this podcast. Artwork and show content belong to their owners. Full legal notice.
Explore this show Podcast research with Pod Engine
