Subvolts take Human judgment is the hidden engine driving AI progress, not just automated processes. Mar 9, 2026 - Developers
- Learn how human evaluation is crucial for improving AI models and discover alternatives to traditional benchmarks.
- Business
- Understand the importance of real-world feedback for AI product development and how to measure model effectiveness.
This interview provides valuable insights into the practical evaluation methods used by AI labs, featuring a key platform in the ecosystem.
#325 Phelim Brady: Why AI's Future Depends on Human Judgement
Audio streams from Craig S. Smith. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Eye on AI episode #325 features Craig Smith interviewing Phelim Bradley, co-founder and CEO of Prolific. They discuss the critical role of human judgment in developing and evaluating AI systems. The conversation highlights how real human feedback is becoming essential for measuring AI performance beyond traditional benchmarks, especially as AI models evolve.
Imagine AI systems are like students. Even with smart teachers (AI developers), they need real people to test their work and give feedback. This episode explains why those human testers are super important for making AI better and more reliable for everyone.
This episode is for anyone interested in the practical realities of AI development. It explains why human input remains vital, even as AI advances. Listeners will learn about the limitations of current AI benchmarks and how platforms like Prolific bridge the gap between AI labs and human evaluators. This is relevant to frontier labs like OpenAI, Google Gemini, Anthropic, Microsoft Copilot, Meta AI, and xAI Grok, as they all rely on robust evaluation methods to improve their models.
- HostCraig S. Smith
- GuestPhelim Bradleyco-founder and CEO, Prolific
- Guest: Phelim Bradley, co-founder and CEO of Prolific
- Show: Eye on AI
- Episode Number: #325
- Topic: The hidden human layer behind modern AI and the reliance on human judgment for evaluation.
- Key Issue: Traditional benchmarks are becoming less reliable for measuring AI model performance.
AI often looks fully automated. But behind the scenes, a huge amount of human judgment is shaping how these systems actually work.
In this episode, Craig Smith speaks with Phelim Bradley, co-founder and CEO of Prolific, a platform that connects millions of real people with researchers and AI labs to evaluate and improve AI systems.
They explore the hidden human layer behind modern AI, why traditional benchmarks are becoming less reliable, and why AI companies increasingly rely on real human feedback to measure model performance in the real world.
Phelim also explains how demographic differences influence how models are evaluated, why human judgment remains critical even as AI improves, and how the collaboration between humans and AI will shape the next phase of development.
This conversation reveals the human backbone behind today's AI systems.
Stay Updated:
Craig Smith on X: https://x.com/craigss
Eye on A.I. on X: https://x.com/EyeOn_AI
(00:00) Preview and Intro
(02:45) Founding Prolific And Early Pain Points
(06:30) From Mechanical Turk To Representativeness
(09:55) Academic Research And AI Use Cases Split
(13:40) Vetting Real Participants And Fighting Fraud
(17:45) Scale, Community Growth, And Talent Mix
(22:00) High-Complexity Projects Over Commoditised Labeling
(26:40) Measuring Model Persuasion With Live Conversations
(30:20) Demographic-Aware Model Preference Benchmarks
(34:10) The Rise Of Human Evaluation Over Benchmarks
(38:00) Enterprise Model Choice And Continuous Evaluation
(42:00) Why Humans Won't Disappear From The Loop
- Benchmarks
- Craig S. Smith
- Standard tests used to measure how well AI systems perform tasks.
- Commoditised Labeling
- Phelim Bradley
- Basic, repetitive data tagging tasks that have become common and less valuable.
55 min42 min51 min34 minUkraine's Secret Weapon: The Points System Winning the Drone War | Andrii Hrytseniuk
WithAndrii Hrytseniuk
Same guests, other shows
24 minWhy Humans Are Still Powering AI [Sponsored]
WithPhelim Bradley
Questions people ask
- What is Prolific?
- Prolific is a platform that connects millions of people with researchers and AI labs to evaluate and improve AI systems.
- Why are traditional AI benchmarks becoming less reliable?
- Traditional benchmarks are becoming less reliable as AI systems become more sophisticated, necessitating real-world human feedback for accurate performance measurement.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Craig S. Smith.