Subvolts take AI labs diverge on AGI paths; benchmarks mislead, human evaluation matters. Dec 15, 2025 - Developers
- Learn about the pitfalls of benchmark optimization and the importance of robust evaluation infrastructure for building better models.
- Business
- Understand why current AI measurement is flawed and how frontier labs' divergent strategies might impact future AI capabilities.
This interview features the CEO of a key data infrastructure provider for major AI labs, offering candid insights into development realities.
Ep 80: CEO of Surge AI Edwin Chen on Why Frontier Labs Are Diverging, RL Environments & Developing Model Taste
Audio streams from by Redpoint Ventures. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Unsupervised Learning Ep 80 features Edwin Chen, CEO of Surge AI, discussing the divergence in frontier AI labs' approaches to AGI. Chen criticizes optimizing for benchmarks like LMArena, highlighting issues with data quality and measurement. The conversation also covers the importance of human evaluations, the rise of RL environments, and challenges in model training.
This episode talks about how top AI companies are building their smart systems. The CEO of a company that helps train AI explains that focusing only on easy tests can be misleading. He shares why real human feedback is crucial for making AI truly better.
This episode offers insights into the practical challenges of AI development beyond public benchmarks. Chen, whose company provides data infrastructure for major AI labs, explains why current measurement methods are flawed and how labs are taking different paths to AGI. It's valuable for AI researchers, developers, and business leaders tracking the real-world progress and strategic decisions of frontier AI companies.
- HostJacob Effron
- HostJordan Segall
- GuestEdwin Chenfounder and CEO, Surge AI
- Guest: Edwin Chen, CEO of Surge AI
- Show: Unsupervised Learning
- Episode: Ep 80
- Length: 48 minutes
- Benchmark Criticism: Optimizing for LMArena is 'basically optimizing for clickbait'.
- Model Regression: One frontier lab's models regressed for 6-12 months without anyone knowing.
Edwin Chen is the founder and CEO of Surge AI, the data infrastructure company behind nearly every major frontier model. Surge works with OpenAI, Anthropic, Meta, and Google, providing the high-quality data and evaluation infrastructure that powers their models.
Edwin reveals why optimizing for popular benchmarks like LMArena is "basically optimizing for clickbait," how one frontier lab's models regressed for 6-12 months without anyone knowing, and why the industry's approach to measurement is fundamentally broken. Jacob and Edwin discuss what actually makes elite AI evaluators, why "there's never going to be a one size fits all solution" for AI models, and how frontier labs are taking surprisingly divergent paths to AGI.
(0:00) Intro
(0:56) The Pitfalls of Optimizing for LMArena
(4:34) Issues with Data Quality and Measurement
(9:44) The Importance of Human Evaluations
(13:40) The Rise of RL Environments
(17:21) Challenges and Lessons in Model Training
(19:59) Silicon Valley's Pivot Culture
(23:06) Technology-Driven Approach
(24:18) Quality Beyond Credentials
(27:51) Impact of Scale Acquisition
(28:35) Hiring for Research Culture
(30:48) Divergence in AI Training Paradigms
(34:16) Future of AI Models
(39:32) Multimodal AI and Quality
(43:44) Quickfire
With your co-hosts:
@jacobeffron
- Partner at Redpoint, Former PM Flatiron Health
@patrickachase
- Partner at Redpoint, Former ML Engineer LinkedIn
@ericabrescia
- Former COO Github, Founder Bitnami (acq’d by VMWare)
@jordan_segall
- Partner at Redpoint
Jacob is an AI investor at Redpoint Ventures. He's led Redpoint's investments in companies like Abridge, Physical Intelligence & Legora. Follow Jacob on Twitter (@jacobeffron).
On Unsupervised Learning we probe the sharpest minds in AI in search for the truth about what's real today, what will be real in the future and what it all means for businesses and the world. If you're a builder, researcher or investor navigating the AI world, this podcast will help you deconstruct and understand the most important breakthroughs and see a clearer picture of reality. Subscribe to this show to stay up to date on our latest episodes.
- AGI
- Edwin Chen
- Artificial General Intelligence, or AI that can perform any intellectual task a human can.
- LMArena
- Edwin Chen
- A popular benchmark used to evaluate and compare AI language models.
- RL Environments
- Jacob Effron, Jordan Segall
- Simulated settings where AI agents learn by trial and error, receiving rewards or penalties.
Same guests, other shows
56 minThe Startup Powering The Data Behind AGI
WithEdwin Chen
Questions people ask
- Why is optimizing for benchmarks like LMArena problematic?
- Optimizing for popular benchmarks like LMArena is described as 'basically optimizing for clickbait,' suggesting it doesn't reflect true AI capability.
- What are the main issues with current AI model measurement?
- The industry's approach to measurement is fundamentally broken, with issues in data quality and a reliance on flawed metrics.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to by Redpoint Ventures.