It Depends: Understanding Psychometrics for Testing + Assessment, by Dr. Amanda Dainis + Tim Burnett
It Depends: Understanding Psychometrics for Testing + Assessment
5.0from 5 ratings
- 12
- Episodes
- 5
- Ratings
- 2023
- First episode
About It Depends: Understanding Psychometrics for Testing + Assessment, by Dr. Amanda Dainis + Tim Burnett
This podcast makes the complex world of psychometrics easy to understand. Tim Burnett and Dr. Amanda Dainis mix their deep knowledge with a bit of fun, aiming to make the topic enjoyable for everyone. They explore how tests are made and their real-life uses, inviting listeners to join the conversation. Whether you're new to psychometrics or already know a lot, this podcast has something for you. A Podcasting Member of the Test Community.Network
- Publisher
- It Depends: Understanding Psychometrics for Testing + Assessment
- Category
- education
- Language
- en
- Explicit
- No
- First episode
- 14 Dec 2023
- Latest episode
- 21 Apr 2026
Latest episodes
12 episodes in the feed.
21 Apr 2026
More Great Content
Do you want more great content? Subscribe to the Test Community Network Podcast: https://open.spotify.com/show/3bqFkMhO2IiwWtlaEMCPjT?si=Jq8uhi1hQ_21xxC2bzU7_Q (https://open.spotify.com/show/3bqFkMhO2IiwWtlaEMCPjT?si=Jq8uhi1hQ_21xxC2bzU7_Q)
29 Nov 2024
It Depends Episode 10: Criterion, Norm, CAT & LOFT
Dr. Amanda Dainis and Tim Burnett explore the differences between criterion referenced, norm referenced, CAT, and LOFT testing, offering insights into effective assessment implementation.
20 Sept 2024
Ep9 - Test Centre V Online Proctoring
This episode of the It Depends Podcast compares online proctoring and test centres for assessment delivery. Amanda and Tim explore the advantages and challenges of both methods, discussing candidate experience, technology reliability, security concerns, and cost factors. They debate the misconceptions surrounding each approach and emphasise the importance of considering an organisation's specific needs and budget constraints. The conversation also touches on future trends, including the impact of AI and the likelihood of both methods coexisting to meet diverse testing requirements. Overall, the episode highlights that the choice between online invigilation and test centres often "depends" on various factors unique to each testing programme. Key questions answered: Which method is preferred or better for candidates? How do the security concerns compare between online proctoring and test centers? What does the future hold for online proctoring and test centers? Topic Keywords: Proctoring, Invigilation, Item banking, Psychometrics, Certification, Security, Cheating, Accessibility, Test centers Bonus Question: What is the future of online proctoring and test centres. Find the answer here: https://www.testcommunity.network/podcast-it-depends Register for Office Hours with Amanda Dainis and Company here: https://www.dainisco.com/ Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ Tim Burnett https://www.linkedin.com/in/tburnett/ Ask a question: https://www.testcommunity.network/podcast-it-depends
14 Aug 2024
Ep8 - Testing on a Budget
This episode of the It Depends Podcast explores testing on a budget, from test development to secure delivery. Amanda and Tim discuss testing on a budget, exploring various ways organisations can implement and maintain testing programs without necessarily investing in expensive, large-scale platforms. Key questions answered: Are big vendor platforms always the best option for your organisation if you're testing on a budget? Is it possible to do item banking on a budget? Can you do secure test delivery on a budget? Topic Keywords: Item banking, test security, remote proctoring, budget testing solutions, data analysis, operational validity, accreditation requirements, scalability, hidden costs, licensing. Bonus Question: If you need a bigger platform, what tips are there to reduce the price? Find the answer here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Register for Office Hours with Amanda Dainis and Company here: https://www.dainisco.com/ (https://www.dainisco.com/) Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett https://www.linkedin.com/in/tburnett/ (https://www.linkedin.com/in/tburnett/) Ask a question: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends)
16 Jul 2024
Ep 7 - Performance Based Testing
This episode of the It Depends Podcast explores performance-based testing in professional certification and licensure. Amanda and Tim discuss the benefits and challenges of performance tests compared to traditional knowledge-based exams, covering various assessment types, implementation considerations, costs, and future trends in performance-based assessment. Key questions answered: What is performance-based testing? What are some advantages of performance tests What challenges come with performance-based tests? Topic Keywords: Psychometrics • Validity • Reliability • Item Response Theory • Job Task Analysis • Cut Scores • Performance-Based Assessment • Test Security • Equating • Bias and Fairness Bonus Question: What future trends do you see in performance testing and how should organizations prepare to adapt to these changes? Find the answer here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett https://www.linkedin.com/in/tburnett/ (https://www.linkedin.com/in/tburnett/) Ask a question: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends)
20 Jun 2024
Episode 6 - Exam Maintenance
Top Episode Takeaways: 📈 Test maintenance involves regularly monitoring test and item performance statistics to ensure content remains current and secure, and should be conducted annually at minimum or as frequently as monthly for large programs. 🔍 The more frequently a test is administered, the more often maintenance should be performed, including reviewing key test-level statistics, item-level metrics, and qualitative candidate feedback to proactively identify issues. 🛠️ Typical maintenance steps include analyzing test passing rates, reliability, item difficulty, discrimination, and distractor performance, investigating qualitative trends, and engaging subject matter experts to replace problematic items following standard development procedures. Bonus Question: Can you list 3 key flags or indicators that would alert you to a problem with the maintenance of a test? Find the answer here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett https://www.linkedin.com/in/tburnett/ (https://www.linkedin.com/in/tburnett/) Ask a question: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends)
12 May 2024
Ep5: Test Blueprints
Top Episode Takeaways: 📐 Test blueprints provide the overall design and structure for an assessment. They define what content areas the test will cover and what proportion of the test each content area should comprise. Having a blueprint helps ensure the test adequately measures the intended knowledge and skills in a balanced way that reflects their importance. 🎯 Blueprints should be created for most types of assessments, including certification/licensure exams, educational tests, and even workplace skills tests. The blueprint is developed after conducting a job task analysis, curriculum review, or other process to determine the key knowledge and skills to be assessed. Subject matter experts should be involved in validating the blueprint. ✅ A good test blueprint will cover all the major content areas without overemphasizing or underemphasizing any topics, allocate questions to each content area proportionately to their importance, be validated by representative subject matter experts to ensure it reflects real-world requirements, be transparently shared with stakeholders so they understand what the test measures, and be updated periodically as the field evolves to ensure the test remains relevant and valid. Bonus Question: Is there a particular platform or technology that you like to use when creating test blubprints? Find the answer here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett https://www.linkedin.com/in/tburnett/ (https://www.linkedin.com/in/tburnett/) Ask a question: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends)
18 Apr 2024
Ep4: Exploring Item Types
Top Episode Takeaways: 📝 Multiple choice questions are the most common item type, but they can be done well or poorly. A good multiple choice question takes effort to create but can measure a wide range of knowledge and skills. 🎥 Technology-enhanced items (e.g., audio/video clips, drag-and-drop) can make assessments more engaging, but they should be used purposefully to measure relevant skills, not just for the "wow" factor. 🚢 Performance-based assessments, such as simulations, are the gold standard for measuring practical skills but can be resource-intensive. The key is finding the right balance based on the content, resources, and scoring considerations. Episode Keywords: Item types, Multiple choice, Technology-enhanced items, Performance-based tests, Practical exams, Simulations, Rubrics, Scoring, Bias, Psychometricians Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett https://www.linkedin.com/in/tburnett/ (https://www.linkedin.com/in/tburnett/) Ask a question: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends)
22 Mar 2024
Ep3: Cut Scores Explained
Top Episode Takeaways: ✂️ Cut scores are important decision points in criterion-referenced tests that categorize test takers as passing or failing. They should be set thoughtfully using a defensible process, not arbitrarily (e.g. always using 70%). Setting the cut score too high or low can have significant consequences. 📏 The Angoff method is a common way to set cut scores. It involves having subject matter experts review test questions and estimate what percentage of minimally competent test takers would answer each question correctly. This provides a way to link the cut score to the test content and the minimum performance standard for the job or domain being tested. ⚖️ Cut scores may need to be revisited and adjusted over time if the test content changes substantially. Small tweaks may not require changing the cut score, but replacing many items with ones that perform differently in terms of difficulty may necessitate an adjustment to keep the cut score fair and aligned with the intended minimum competency standard. Ongoing evaluation is important. Episode Keywords:Validity, Reliability, Construct, Cut score, Angoff method, Item analysis, Fairness, Rubric, Feedback, Blueprint. Connect with us: Amanda Dainis, PhD, MPA https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/ (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett https://www.linkedin.com/in/tburnett/ (https://www.linkedin.com/in/tburnett/) A proud supporter of the https://www.testcommunity.network
20 Feb 2024
Special Episode 1 - Vendor Shopping
Top Episode Takeaways: 🤝 Tactics for negotiation - Ask vendors about integration costs upfront, and use your accreditation status as leverage or potential for publicity. Sometimes, size does matter. Too big and you're one among many, too small and they can't keep up. 📁 Importance of owning your data and access - Ensure you can export items, stats, and raw response data. This relationship may not last forever, your business may change, and the provider might be purchased, what happens then? Be clear you can access data and think beyond onboarding. 📊 Psychometric reporting - If you're going to be doing your psychometric reporting, then make sure you have access to the data, the zeros and ones. If they have neat reporting tools built in, make sure they work for you and not just for that one big client they built it for. Listen to the Bonus Question: What is your favourite platform, and get the full transcript here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Learn about: Platform, Vendor, Item bank, Statistics, Psychometrics, Reporting, Standard setting, Item types, AI / Artificial intelligence, Legal defensibility, Test development, Candidate experience, Test delivery, Remote proctoring, Accreditation Connect with us: Amanda Dainis, PhD, MPA (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett (https://www.linkedin.com/in/tburnett/) A proud supporter of the https://www.testcommunity.network (https://www.testcommunity.network/)
30 Jan 2024
Ep2 - JTAs (Job Tasks Analysis), Unravelling the Essentials of Effective Test Development
Top Episode Takeaways: 🧠 Importance of Job Task Analysis (JTA) in Psychometrics: The podcast emphasizes the critical role of JTA in creating and updating certification exams. JTA helps ensure that exam questions are relevant and representative of the actual knowledge, skills, and abilities required for the job. 🤝 Role of Subject Matter Experts (SMEs): SMEs are vital in the JTA process. Their input helps form a comprehensive understanding of the job role. The podcast highlights the challenges and strategies in recruiting and managing SMEs, and the importance of diverse representation from various geographical locations and job roles. 📊 Survey and Analysis in JTA: A significant part of JTA involves conducting a survey among industry professionals to validate the competencies and tasks identified by SMEs. This data is then analyzed to create an exam blueprint, guiding the distribution of questions across different job-related topics. Get the full transcript here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Learn about: Psychometrics, Job Task Analysis (JTA), Certification Exams, Subject Matter Experts (SMEs), Exam Blueprint, Knowledge, Skills, and Abilities (KSAs), Test Development, Accreditation, Item Writing, Survey Analysis, Validation, Cut Scores, Legal Challenges, Psychometricians Connect with us: Amanda Dainis, PhD, MPA (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett (https://www.linkedin.com/in/tburnett/) A proud supporter of the https://www.testcommunity.network (https://www.testcommunity.network/)
14 Dec 2023
Ep1 - Psychometrics for the Non-Psychos: A Friendly Introductory Journey
Top Episode Takeaways: 🤝 Meet the Experts: Discover the unique backgrounds and expertise of hosts Tim Burnett and Dr. Amanda Dainis. 📊 Behind the Tests: Uncover the intricate process of test development and what makes a good test. 🌟 Breaking Stereotypes: Learn how psychometrics is more than just dry statistics – it's a field filled with creativity and real-world impact. Get the full transcript here: https://www.testcommunity.network/podcast-it-depends (https://www.testcommunity.network/podcast-it-depends) Learn about: Psychometrics, Test Development, Assessment, Educational Testing, Validity, Reliability, Data Analysis, Measurement Theory, Certification, Statistical Analysis Connect with us: Amanda Dainis, PhD, MPA (https://www.linkedin.com/in/amanda-dainis-phd-mpa-012b94119/) Tim Burnett (https://www.linkedin.com/in/tburnett/) A proud supporter of the https://www.testcommunity.network (https://www.testcommunity.network/)
Reach and audience
Public platform figures. Ratings count people who left a rating, not total listeners.
- Apple Podcasts (US)
- 5.0 / 5
- 5 ratings
Host of It Depends: Understanding Psychometrics for Testing + Assessment, by Dr. Amanda Dainis + Tim Burnett?
Claim your podcast to manage its listing and keep your show details accurate.
Pod Engine is an independent podcast discovery and analytics service and is not affiliated with or endorsed by this podcast. Artwork and show content belong to their owners. Full legal notice.
Explore this show
Podcast research with Pod Engine