BeyondQuality

About BeyondQuality

The Beyond Quality Podcast brings QA/QE professionals together to explore current research from our collaborative community. Each month, we discuss open projects, share insights, and highlight how collaborative work is pushing the craft of quality forward.

Publisher

Vitaly Sharovatov

Category

technology

Language

en

Explicit

No

First episode

9 Sept 2025

Latest episode

18 Sept 2026

Latest episodes

13 episodes in the feed.

1. 

   18 Sept 2026

   Episode 13: The role of quality in AI governance

   What happens when an AI agent looks for an API key and deletes a production database, when it wasn’t supposed to? As quality professionals, governance wasn’t a topic that particularly interested us. Most organizations have glaring gaps in governance that were ignored thus far. In this episode with Brijesh Deb, Vitaly and Anupam dig deeper into governance and conclude how with AI, these governance gaps become too big to ignore. What’s more? Fixing that gap is a perfect opportunity for quality professionals to establish greater relevance in the era of AI-powered software development. Brijesh offered a 7-point framework to think about AI Governance: 1. Work intent & context: Is the AI still doing the work we think it is doing, for the users and conditions we actually have now? 2. Autonomy boundaries: What can it decide or do without intervention, and is that autonomy still justified? 3. Memory & state: What persists, what can become stale or hidden, and can people inspect and correct it? 4. Permissions & tools: Does access reflect the task, or has authority accumulated because it was convenient? 5. Evidence & evaluation: Does the strength of evidence match the consequence of being wrong? 6. Human review: Can a person realistically detect and stop a harmful action, or is a human merely present on paper? 7. Monitoring & learning: Are we evaluating the living workflow and changing controls when reality moves? He aims to take his inquiry further into building an evidence to decision loop: a basis to collect the right kind of evidence so that stakeholders can make informed decisions with regard to software, where AI is involved. Links: Discussion Thread: https://github.com/BeyondQuality/beyondquality/discussions/43 (https://github.com/BeyondQuality/beyondquality/discussions/43) Vitaly’s AI governance research artifact: https://github.com/BeyondQuality/beyondquality/blob/main/research/quality-governance/quality-governance.md (https://github.com/BeyondQuality/beyondquality/blob/main/research/quality-governance/quality-governance.md) Brijesh LinkedIn: https://www.linkedin.com/in/debbrijesh/ (https://www.linkedin.com/in/debbrijesh/) PocketOS incident: https://www.fastcompany.com/91533544/cursor-claude-ai-agent-deleted-software-company-pocket-os-database-jer-crane (https://www.fastcompany.com/91533544/cursor-claude-ai-agent-deleted-software-company-pocket-os-database-jer-crane)

2. 

   14 Aug 2026

   Episode 12: AI governance: automated decisions and human responsibility

   AI can now make or support decisions that were once made only by humans. But there is one thing AI cannot take on: responsibility. When an AI system recommends, approves, rejects, prioritises or acts, a human or organisation still needs to be accountable for the outcome. Meaningful human oversight requires more than putting a person at the end of the process and asking them to click the Approve button. The person needs access to the right information: what AI decided, why, what data influenced the decision, what uncertainty exists, and what risks or alternatives were considered. In this episode, we explore the gap between AI decision-making and human responsibility. We discuss what humans need in order to meaningfully sign off on AI-supported decisions, what reliable oversight could look like, and whether organisations are designing their AI systems around accountability. Links: - Enquiry discussion on BeyondQuality: https://github.com/BeyondQuality/beyondquality/discussions/47 (https://github.com/BeyondQuality/beyondquality/discussions/47) - AI Era Testing research: https://beyondquality.org/research/ai-era-testing/ (https://beyondquality.org/research/ai-era-testing/)

3. 

   24 Jul 2026

   Episode 11: Can LLM Judges go beyond automated testing?

   LLM outputs are hard to automatically test. You can’t use assertions for verifying a software that produces varied outputs for the same inputs that may nevertheless be acceptable. Evaluation of AI systems using human domain experts might work for a few cases, but this doesn’t scale. We need automation of judgement and not merely string comparison. But once you do automate judgement at scale, can you go from verification to validation? Can you move from from testing an AI product to improving it? This episode centers around Anupam’s enquiry on LLM judges. Hosts Maryia, Vitaly and Anupam discuss the two hypotheses he presented in this enquiry: 1. An LLM-Judge must be calibrated by one or more domain-experts for it to be useful 2. A calibrated LLM judge could go beyond being a means for catching defects, and could actually be used as a calibrated instrument for continuous improvement of an AI-System They further talk about why calibration is needed, how teams should perform it, and what factors trigger recalibration of am LLM Judge. Links: - Discussion on BeyondQuality: https://github.com/BeyondQuality/beyondquality/discussions/40 (https://github.com/BeyondQuality/beyondquality/discussions/40) - Recording of Anupam’s talk on LLM judge calibration: https://www.youtube.com/watch?v=E6B7lrrhrnM (https://www.youtube.com/watch?v=E6B7lrrhrnM) - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/pdf/2306.05685 (https://arxiv.org/pdf/2306.05685)

4. 

   27 May 2026

   Episode 10: deep dive in AI era testing research

   With developers using AI, how do quality professionals deal with reviewing increasing amounts of work to review? How can we keep pace with AI-accelerated coding? In this episode, we spoke about two different kinds of quality teams – proactive and reactive. We then explore the coordination and rework challenges that AI accelerated development poses to each of these teams. We finally end with a sobering prediction: reactive quality practices that were already wasteful in the pre-AI era will now become unsustainable. Research: https://github.com/BeyondQuality/beyondquality/tree/main/research/ai-era-testing (https://github.com/BeyondQuality/beyondquality/tree/main/research/ai-era-testing) Sources: - B. W. Boehm, "Software Engineering”, 1976 (https://selab.netlab.uky.edu/homepage/boehm-sw-eng-paper.pdf (https://selab.netlab.uky.edu/homepage/boehm-sw-eng-paper.pdf)) - Stafford Beer, "Brain of the firm", https://www.amazon.com/Brain-Firm-Stafford-Beer/dp/047194839X (https://www.amazon.com/Brain-Firm-Stafford-Beer/dp/047194839X)

5. 

   16 Apr 2026

   Episode 9: AI, Testing and DORA with Lisa Crispin

   Vitaly, Anupam, and Maryia sit down with Lisa Crispin (independent consultant, DORA community guide, co-author with Janet Gregory of the Agile Testing books). Lisa watched the industry fire testers when automation arrived in the early 2000s. The same panic is repeating with AI, but non-deterministic AI output demands more quality thinking, not less. Big batches never worked well; now they stop working entirely. Testing-after never worked well; now it can't work at all. DORA's 2025 research captures the pattern: AI is an amplifier. Whatever teams were already doing, good or bad, gets scaled up. The capabilities that correlate with high-performing AI-augmented teams are the same ones that made human teams high-performing before AI. But the industry still sells AI as a way to make individual developers 10x more productive. The result: software delivery instability is rising even as individual productivity goes up, because code reaches production badly tested and the rework drives burnout. Vitaly thinks AI is a great collaboration amplifier: the typing bottleneck is gone, freeing cross-functional teams to iterate faster, together defining why and how the software is to be built and tested. Links: Lisa's Linkedin: https://www.linkedin.com/in/lisacrispin/ (https://www.linkedin.com/in/lisacrispin/) BQ research: https://github.com/BeyondQuality/beyondquality/blob/main/research/ai-era-testing/analysis.md (https://github.com/BeyondQuality/beyondquality/blob/main/research/ai-era-testing/analysis.md) DORA AI Capabilities report: https://cloud.google.com/resources/content/2025-dora-ai-capabilities-model-report (https://cloud.google.com/resources/content/2025-dora-ai-capabilities-model-report) Sponsors: Qase https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy (https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy) Regent https://regent.se (https://regent.se) TestSolutions https://testsolutions.de/en/focus/ai-testing (https://testsolutions.de/en/focus/ai-testing)

6. 

   28 Mar 2026

   Episode 8: AI generates code. Who carries the understanding?

   Vitaly, Anupam, and Maryia sit down with Liliia Abdulina — Head of QA at JetBrains and co-researcher on the BeyondQuality AI-era testing study — who came to this topic the way most useful research starts: with a real problem on her hands. Developers at JetBrains were accelerating with AI, QA was becoming the bottleneck, and she wanted to understand why. The conversation that follows makes the case that this bottleneck isn't new — managers were already complaining "why is testing delaying us" long before AI. What AI did was make a structural problem impossible to ignore. When humans wrote code, understanding came for free: the person who wrote it carried the mental model. Now code is generated at machine speed and nobody carries it. The group calls this cognitive debt — distinct from technical debt, and harder to see. But the subtler casualty is learning. When a tester finds a bug, the developer carries that knowledge forward and writes fewer similar bugs next cycle. Agents don't learn across sessions, so the loop breaks. The industry's response has been to flood the market with AI inspection tools, because inspection is easy to automate. Prevention — which costs less — requires domain understanding that agents don't have, and nobody is building tools for that. Links: 1. Lilia's Linkedin: https://www.linkedin.com/in/liliiaabdulina/ (https://www.linkedin.com/in/liliiaabdulina/) 2. research discussion https://github.com/BeyondQuality/beyondquality/discussions/28 (https://github.com/BeyondQuality/beyondquality/discussions/28) and current artifact https://github.com/BeyondQuality/beyondquality/blob/main/research/ai-era-testing/analysis.md (https://github.com/BeyondQuality/beyondquality/blob/main/research/ai-era-testing/analysis.md) 3. https://devblogs.microsoft.com/dotnet/ten-months-with-cca-in-dotnet-runtime/ (https://devblogs.microsoft.com/dotnet/ten-months-with-cca-in-dotnet-runtime/) 4. https://arxiv.org/pdf/2603.22106v2 (https://arxiv.org/pdf/2603.22106v2) Sponsors: 1. Qase https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy (https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy) 2. Regent https://regent.se (https://regent.se) 3. AnuKrit https://anukrit.de (https://anukrit.de)

7. 

   27 Feb 2026

   Episode 7: "On trust"

   Maryia, Vitaly, and Anupam sit down with Asya Isakova — an organisational psychologist who's been studying how people function at work since 2007 — to explore her BeyondQuality research on trust in teams. The most counterintuitive finding: trust matters most in hierarchical organisations, precisely the ones that tend to destroy it with KPIs and approval chains. In high-hierarchy teams, subordinates decide what information to share based on whether they trust their manager — and if they don't, the entire decision chain corrupts. The conversation unpacks how lack of trust works like a tax: full workdays wasted on two-minute tasks, teams of 20 writing reports instead of working, $15 purchase orders going through five approvals. Why vulnerability — the core of trust — is the hardest thing for leaders to accept. And why preventive quality work like pair programming or shifting testing left simply cannot start without trust: you can't prove a prevented bug, so if nobody trusts you, nobody lets you try. Links: 1. Asya's Linkedin: https://www.linkedin.com/in/asyaisakova/ (https://www.linkedin.com/in/asyaisakova/) 2. Asya's research proposal: https://github.com/BeyondQuality/beyondquality/discussions/26 (https://github.com/BeyondQuality/beyondquality/discussions/26) 3. Organizational Justice meta-analysis: https://psycnet.apa.org/buy/2013-06399-001 (https://psycnet.apa.org/buy/2013-06399-001) 4. Trust factors and performance review: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2023.1081086/full (https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2023.1081086/full) 5. Team factors and trust: https://onlinelibrary.wiley.com/doi/abs/10.1002/job.2436 (https://onlinelibrary.wiley.com/doi/abs/10.1002/job.2436) Sponsors: 1. Qase https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy (https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy) 2. Regent https://regent.se (https://regent.se) 3. AnuKrit https://anukrit.de (https://anukrit.de)

8. 

   25 Jan 2026

   Episode 6: "on testing economics"

   Maryia, Vitaly, and Anupam sit down with returning guest Mike Jarred — an exceptionally seasoned testing consultant — to unpack Vitaly’s "Economics of Testing" research and how it proposes we talk about quality. The conversation starts with a familiar problem: quality and risk are hard to "make tangible", so testing is too often reduced to outputs and activity. From there, it becomes a practical discussion of how to translate quality work into the language leadership (and other functions) actually uses: the economical language of risks, value and money. Links: 1. Mike's Linkedin: https://www.linkedin.com/in/mikejarred/ (https://www.linkedin.com/in/mikejarred/) 2. Vitaly's research on economics of testing: https://beyondquality.org/research/testing_economics/testing_economics (https://beyondquality.org/research/testing_economics/testing_economics) Sponsors: 1. Qase https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy (https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy) 2. Regent https://regent.se (https://regent.se) 3. AnuKrit https://anukrit.de (https://anukrit.de)

9. 

   4 Dec 2025

   Episode 5: "On laziness, motivation and management"

   Maryia and Vitaly sit down with Maaret Pyhäjärvi — an exceptionally seasoned tester, manager, consultant and speaker whose work has influenced teams and communities around the world. What begins as a conversation about a persistent managerial myth — the belief that employees are inherently lazy — quickly evolves into a discussion of the forces that shape motivation at work. We also we ventured the broader themes of professional ethics, education in tech, leadership models, and what it really means to create environments where people can thrive. Links: 1. Maaret's Linkedin: https://www.linkedin.com/in/maaret/ (https://www.linkedin.com/in/maaret/) 2. Vitaly's research on laziness: https://beyondquality.org/research/laziness (https://beyondquality.org/research/laziness) Sponsors: 1. Qase https://qase.io/ (https://qase.io/) 2. Regent https://regent.se (https://regent.se) 3. AnuKrit https://anukrit.de (https://anukrit.de)

10. 

    14 Nov 2025

    Episode 4: "Standing out in a tough job market"

    Maryia and Anupam spoke to Hannah Reynolds and Mike Jarred, who bring between them more than 3 decades of recruitment and hiring experience. You’ll learn how you can stand out in today’s job-market and dodge pitfalls that most candidates fall prey to. Most importantly, you’ll receive invaluable guidance on how to communicate the value you deliver, and cultivate relationships that will supercharge your career in the long-run. Links: 1. Hannah’s LinkedIn: https://www.linkedin.com/in/hannah-reynolds-b0b579168/ (https://www.linkedin.com/in/hannah-reynolds-b0b579168/) 2. Mike’s LinkedIn: https://www.linkedin.com/in/mikejarred/ (https://www.linkedin.com/in/mikejarred/) Sponsors: 1. Qase https://qase.io/ (https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy) 2. Regent https://regent.se (https://regent.se) 3. AnuKrit https://anukrit.de (https://anukrit.de)

11. 

    17 Oct 2025

    Episode 3: "LLMs, RAG, Ragas and Testing"

    In this episode Vitaly, Maryia and Anupam engage with guest Rahul Verma to discuss the evolving landscape of testing in the context of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG). The conversation explores the challenges of testing LLMs, the importance of understanding the principles behind these technologies, and the need for testers to contribute to the development of frameworks and benchmarks. The discussion emphasizes the significance of approaching LLMs critically, rather than blindly trusting them or, conversely, rejecting them altogether. Links: 1. Anupam's research on RAG evaluation using RAGAS: https://beyondquality.org/research/rag-evaluation (https://beyondquality.org/research/rag-evaluation) 2. Rahul's book "The Last Book on Testing": https://www.amazon.com/Last-Book-Testing-Rahul-Verma-ebook/dp/B0DZRJFX65 (https://www.amazon.com/Last-Book-Testing-Rahul-Verma-ebook/dp/B0DZRJFX65) Sponsors: 1. Qase https://qase.io/ (https://qase.io/?utm_source=youtube&utm_medium=beyondquality&utm_campaign=devadvocacy) 2. Regent AB https://regent.se (https://regent.se) 3. AnuKrit https://anukrit.de (https://anukrit.de)

12. 

    26 Sept 2025

    Episode 2: "Cultural studies in testing"

    In this episode, Vitaly, Anupam and Maryia invited Mesut Durukal — a renowned conference speaker with 20+ years in testing, who has worked across multiple countries and environments — explore how cultural differences shape the way we work, test, and collaborate. From Japanese process-driven approaches to Swedish family-first values, we discuss how history, hierarchy, and cultural habits influence communication, quality standards, and team dynamics in IT and beyond. Mesut also shares insights from his ongoing research into the impact of national cultures on testing practices and invites the community to contribute. Links: 1. BeyondQuality community (https://beyondquality.org) 2. Mesut research "how-to contribute": (https://github.com/mesutdurukal/Global-Testing-Practices/blob/main/CONTRIBUTING.md) 3. Mesut linkedin (https://www.linkedin.com/in/mesutdurukal/) 4. Culture map (https://www.amazon.com/Culture-Map-INTL-ED-Decoding/dp/1610392760) book 5. Management Across Cultures book (https://www.cambridge.org/highereducation/books/management-across-cultures/BB7FBBF5BC5B441AE0F3D3EA3817BB7A)

13. 

    9 Sept 2025

    Episode 1: "Why the Community"

    In the very first episode, Vitaly Sharovatov, Anupam Krishnamurthy, and Maryia Tuleika sit down to talk about the why and how behind the Beyond Quality collaborative community. We share the motivation for creating a space where QA/QE professionals can research, prepare talks, and solve problems together; and how this community can help us all save time, improve quality, and push the craft of quality forward.

Reach and audience

Public platform figures. Ratings count people who left a rating, not total listeners.

Spotify

5.0 / 5

2 ratings

Contact BeyondQuality

Guest appearances

Books guests

Based on episode analysis; this does not confirm that the show is currently accepting guests.