Pod Engine
Podcast thumbnail for The Experimentation Edge

The Experimentation Edge

Claim This Podcast

by Growthbook

24 episodes
Updated Daily
Accepts GuestsHas Sponsors

Podcast Overview

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale. Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams. Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Language

🇺🇲

Publishing Since

2/24/2026

Visit WebsiteAppleView on Apple PodcastsRSS Feed

Reach the team behind The Experimentation Edge

Verified contact details for this show aren't on file yet — sign up to get notified when they land.

Sign up for updates

Recent Episodes

Episode thumbnail for The holy grail metric Stitch Fix says every experimenter should chase

July 2, 2026

The holy grail metric Stitch Fix says every experimenter should chase

<p><strong>Summary</strong></p><p>In this episode of The Experimentation Edge, GrowthBook CMO Ashley Stirrup sits down with Nick Beyler, data science manager at Stitch Fix, where he leads the decision and insights team and owns the company's internal experimentation platform. Nick shares why the metric he most wants is the one he can't measure yet, a North Star that predicts a client's long-term value from their earliest behaviors, and why the most impactful experiment learnings tend to come from adoption friction rather than product bugs. He makes the case that if you're only testing winners you're not taking enough risks, explains how guardrails make that risk safe, and looks ahead to a new in-house platform and the promise of agentic AI. It's a practical, statistician's-eye view of experimentation for product managers, data scientists, and engineers building serious testing programs.</p><p><br><strong><br>Chapters</strong></p><p>00:00 Cold open and welcome to the show</p><p>01:45 What Stitch Fix actually does</p><p>04:15 Balancing AI with the human stylist</p><p>05:15 From public policy to the A/B testing adrenaline rush</p><p>07:15 Inside the weekly experimentation review group</p><p>08:45 The AI style assistant and listening to qualitative feedback</p><p>10:45 Why adoption friction beats product bugs</p><p>13:45 Testing for losers and building guardrails</p><p>15:45 Keep rate, successful fixes, and the holy grail metric</p><p>18:15 The new platform and the promise of agentic AI</p><p><br><strong><br>Takeaways</strong></p><ul><li>The most impactful experiment learnings usually come from adoption friction, not product bugs. By the time a big feature reaches A/B testing, it's often already a winner, so the open question is how and where to introduce it.</li><li>A losing test is a finding, not a failure. If every experiment wins, you're not taking enough risk to learn anything new.</li><li>Guardrails and stopping criteria are what make risk-taking safe, especially when the experience is as personal as shopping.</li><li>The most valuable North Star metric is the one you can't measure yet, long-term client value, and causal-inference modeling helps predict it from short-term behavior.</li><li>Quantitative results are only half the story. Direct, qualitative client feedback inside an experiment often reshapes the rollout more than the numbers do.</li></ul><p><strong><br>Connect with the Guest</strong></p><p>LinkedIn: <a href="https://www.linkedin.com/in/nick-beyler-381864119/">https://www.linkedin.com/in/nick-beyler-381864119/</a> <br>Website: <a href="https://www.stitchfix.com">https://www.stitchfix.com</a></p><p><strong>Sponsor</strong><br>GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.</p><p>Go to <a href="http://growthbook.io?utm_source=edge-podcast&amp;utm_medium=podcast&amp;utm_campaign=episode-25">http://growthbook.io</a></p>

Episode thumbnail for What the Expedia Group cannot measure, it cannot ship

July 1, 2026

What the Expedia Group cannot measure, it cannot ship

<p><strong><br>Summary</strong></p><p>Amir Moghaddam, Director of Software Engineering at Expedia Group, joins host Ashley Stirrup on The Experimentation Edge to make the case that measurement is not a reporting step but a gate: what you cannot measure, you cannot ship. Drawing on nearly four years at DoorDash and his current work leading Expedia's air booking platform, Amir explains why he refuses to label experiments winners or losers, how a "failed" pricing test pushed his team toward full personalization, and why a three sided marketplace forces hard trade-offs between competing metrics. The conversation closes on how the same experimentation discipline now applies to shipping and measuring AI. Built for product managers, engineers, data scientists, and growth leaders who care about rigor over opinion.</p><p><strong><br>Chapters</strong></p><p>00:00 Cold open<br>00:50 Meet Amir and the air booking platform at Expedia<br>03:10 DoorDash, growth, and a 70 experiment year<br>04:20 Three kinds of experimentation at Expedia<br>06:30 AI velocity and the new frontier model pace<br>08:30 What you cannot measure, you cannot ship<br>10:45 The DoorDash carousel and the price experiment<br>12:45 The three sided marketplace and competing metrics<br>16:55 There are no losing experiments<br>20:45 Predictability, LLMs, and Expedia's road ahead</p><p><strong><br>Takeaways</strong></p><ul><li>"What you cannot measure, you cannot ship" — if you can't measure an outcome, you can't decide whether it's better, so you're just debating opinions.</li><li>Measurement spans three live dimensions: spend (more with less), speed (sprints instead of quarters), and quality, with guardrail "do no harm" metrics on top.</li><li>There are no losing experiments. A flat result is a signal to either refine the hypothesis or step back and look from a completely different angle.</li><li>DoorDash's price experiment proved price by itself doesn't predict orders. Different customers want different things at different times, which pushed the team toward personalization.</li><li>A three sided marketplace (buyers, merchants, Dashers) makes metrics compete. Running the test is easy; deciding what to optimize when goals conflict is the real work.</li></ul><p><br><strong><br>Connect with the Guest</strong></p><p>LinkedIn: <a href="https://www.linkedin.com/in/amirmoghaddam">https://www.linkedin.com/in/amirmoghaddam</a><br>Website: <a href="https://www.expediagroup.com">https://www.expediagroup.com</a></p><p><strong>Sponsor</strong><br>GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. </p><p>Go to <a href="http://growthbook.io">growthbook.io</a></p>

Episode thumbnail for How Fin (formerly Intercom) went from weeks to hours of analysis using AI

June 30, 2026

How Fin (formerly Intercom) went from weeks to hours of analysis using AI

<p><strong>Summary</strong></p><p>In this episode of The Experimentation Edge, host Ashley Stirrup sits down with Raunak Kumar, senior manager of GTM analytics at Fin (formerly Intercom), to unpack how experimentation actually works when the data is messy and the traffic is thin. Drawing on nearly 12 years in marketing analytics across Atlassian, Stripe, and Fin, Raunak explains how AI tools like Claude Code have collapsed analysis from weeks to hours and freed his team to clear its experiment backlog, why declining organic search traffic and a 5x jump in untagged ChatGPT referrals are forcing teams to rethink attribution, and how the most valuable experiments are often the ones that "lose." From a Jira Service Desk bundling test that won on trials but had to be rolled back, to a Stripe contact form that was quietly blocking real buyers, this conversation is a practical guide for product managers, engineers, data scientists, and growth marketers who want to learn more from every test they run.</p><p><br><strong><br>Chapters</strong></p><p>0:45 Welcome and what the show is about<br>1:45 Raunak's role and 12 years in marketing analytics<br>2:45 How AI and Claude Code changed the analyst's day<br>4:15 LLMs, declining organic traffic, and the 5x ChatGPT jump<br>5:15 Two kinds of experiments at Fin: on page and off page<br>7:15 The Jira Service Desk bundling experiment<br>10:45 Why the trial winner became a rollback<br>11:45 Contextual onboarding turns the loser into a winner<br>14:45 Reading an experiment that loses<br>18:45 What's next: incrementality, connected TV, and testing creative</p><p><br><strong><br>Takeaways</strong></p><ul><li>AI has collapsed marketing analysis from weeks to hours, and the real payoff is a cleared experiment backlog plus analysts who compete on the questions they ask, not the speed they query.</li><li>Organic search traffic is declining as ChatGPT, Gemini's AI mode, and Claude answer buyers in place; Fin saw a 5x rise in ChatGPT referrals, but LLMs don't tag that traffic, so attribution has to be proven through experiments.</li><li>A guardrail metric saved Atlassian from a costly mistake: bundling Jira Service Desk lifted trials more than 50 percent but tanked activation and paid conversion, forcing a rollback.</li><li>A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.</li><li>In low-volume B2B, read losing experiments for sub-segment signal; a "failed" Stripe form simplification revealed the form was blocking legitimate small-business buyers using Gmail.</li></ul><p><br><strong><br>Connect with the Guest</strong></p><p>LinkedIn: <a href="http://linkedin.com/in/raunakkumar1991">http://linkedin.com/in/raunakkumar1991</a><br>Website: <a href="https://fin.ai">https://fin.ai</a></p><p><strong>Sponsor</strong><br>Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.</p><p>Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.</p><p>With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.</p><p>See a demo at <a href="https://www.growthbook.io/">https://www.growthbook.io/</a></p>

24 total episodes available

Deep-dive analytics for The Experimentation Edge

Sign up to explore

Frequently asked questions

Have a different question and can't find the answer you're looking for? Reach out to our support team by sending us an email and we'll get back to you as soon as we can.

What is The Experimentation Edge?

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.

Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.

Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

How often does this podcast release new episodes?

This podcast updates daily.

Where can I listen to this podcast?

This podcast is available on 4 platforms including Apple Podcasts, Spotify, and more. You can also use the RSS feed directly.

Does this podcast accept guests?

Yes, this podcast regularly features guests.

Legal Disclaimer

Pod Engine is not affiliated with, endorsed by, or officially connected with any of the podcasts displayed on this platform. We operate independently as a podcast discovery and analytics service.

All podcast artwork, thumbnails, and content displayed on this page are the property of their respective owners and are protected by applicable copyright laws. This includes, but is not limited to, podcast cover art, episode artwork, show descriptions, episode titles, transcripts, audio snippets, and any other content originating from the podcast creators or their licensors.

We display this content under fair use principles and/or implied license for the purpose of podcast discovery, information, and commentary. We make no claim of ownership over any podcast content, artwork, or related materials shown on this platform. All trademarks, service marks, and trade names are the property of their respective owners.

While we strive to ensure all content usage is properly authorized, if you are a rights holder and believe your content is being used inappropriately or without proper authorization, please contact us immediately at hey@podengine.ai for prompt review and appropriate action, which may include content removal or proper attribution.

By accessing and using this platform, you acknowledge and agree to respect all applicable copyright laws and intellectual property rights of content owners. Any unauthorized reproduction, distribution, or commercial use of the content displayed on this platform is strictly prohibited.