Subvolts take AI benchmarks are broken; test-time compute reveals true model power. Jun 26, 2026 - Developers
- Learn how to better evaluate AI models and understand the implications of compute budgets on performance.
- Business
- Understand how AI capabilities are measured and the potential for more advanced reasoning in AI systems.
This interview with an OpenAI researcher explores fundamental issues in AI evaluation and safety, impacting how models are understood.
Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI Research Scientist Noam Brown
Audio streams from Conviction. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
OpenAI research scientist Noam Brown joins No Priors hosts Sarah Guo and Elad Gil to discuss how AI model evaluation is broken. Traditional static benchmarks do not account for test-time compute, the amount of thinking time allowed. Brown argues that current models can reason for extended periods, impacting how we measure capability, safety, and research.
Imagine judging a student on a test they only have 5 minutes to think about. This episode explains that current AI tests are like that, not giving models enough time to truly show what they know. Giving AI more thinking time could unlock surprising abilities and change how we test AI safety.
This episode is for AI researchers and developers interested in the nuances of model evaluation. It highlights how current benchmarks fail to capture the full potential of AI models, especially as they are given more time to compute. This is directly relevant to frontier labs like OpenAI, Google Gemini, and Anthropic, as it questions the very metrics they use to compare and improve their systems. Understanding test-time compute is crucial for developing more accurate AI safety frameworks and exploring advanced AI capabilities.
- HostSarah Guo
- HostElad Gil
- GuestNoam BrownResearch Scientist, OpenAI
- Guest: Noam Brown, OpenAI Research Scientist
- Show: No Priors
- Length: 36 minutes
- Core Argument: Traditional AI benchmarks are broken because they do not account for test-time compute.
- Potential Capability: Models can reason for weeks or months on complex tasks with sufficient compute.
When a new AI model drops, it’s judged based on a static benchmark grid that doesn’t account for how long the model is allowed to think. How then should we measure a model’s true capability? OpenAI research scientist Noam Brown returns to talk with Sarah Guo about his latest essay on why the AI industry’s traditional benchmark grids are broken, and how large-scale test-time compute is fundamentally changing how models are evaluated. Noam explains how, if properly scaffolded, today’s models can reason for weeks or even months on complex tasks. He also discusses real-world implications of test-time compute, from building poker solver bots to disproving legendary math conjectures. Together, they also unpack the large gaps in current AI safety frameworks, explore the bottlenecks for recursive self-improvement, and look ahead at the future of multi-agent collaboration and global knowledge sharing.
Read more: Implications of Large-Scale Test-Time Compute
Sign up for new podcasts every week. Email feedback to show@no-priors.com
Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @polynoamial | @OpenAI
Chapters:
00:00 – Cold Open
00:43 – Noam Brown Introduction
01:23 – Why Benchmarks Are Broken
04:19 – Compute Budgets and Projections
05:34 – How Long Should Models Think?
06:47 – Benchmark-Maxxing
08:34 – Using Poker Bots as Evals
11:26 – Safety Evals When Model Capability Scales With Budget
14:41 – Release Cycle vs. Agent Runtime
17:06 – Latent Model Capability
20:59 – Limits on Recursive Self-Improvement
27:09 – Large-Scale Multi-Agent Coordination
29:11 – Competition at the Frontier
31:51 – Breaking the Benchmark Grid Equilibrium
33:29 – Why Benchmarks Should be Evaluated by Cost
36:18 – Conclusion
- Test-time compute
- Noam Brown
- The amount of thinking time an AI model gets when it's being tested.
- Benchmarks
- Sarah Guo
- Standardized tests used to measure and compare the performance of AI models.
- Recursive self-improvement
- Elad Gil
- When an AI system can improve its own abilities over time.
39 min1 h 1136 minSame guests, other shows
1 h 20
48 minImage Generation and Visual Intelligence with Black Forest Labs
WithDustin Podell
Questions people ask
- Why are current AI benchmarks flawed?
- Traditional benchmarks use static grids that do not account for the amount of test-time compute, or thinking time, allowed for an AI model.
- What is test-time compute?
- It is the amount of time a model is allowed to think or process information during evaluation, which can significantly impact its performance on complex tasks.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Conviction.