Subvolts take Understand if AI does the right thing for the wrong reason with Apollo Research's new method. Jul 31, 2026 - Developers
- Learn about advanced techniques for testing AI alignment and understanding model motivations beyond surface performance.
- Business
- Understand the challenges in ensuring AI systems behave reliably and ethically, even with complex internal reasoning.
This episode features researchers from Apollo Research discussing new alignment methods developed with OpenAI.
How Researchers Test AI for Hidden Goals — Apollo Research
Audio streams from Machine Learning Street Talk (MLST). Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Machine Learning Street Talk features Apollo Research's Alexander Meinke, Axel Højmark, and Jérémy Scheurer discussing their new research with OpenAI. They explore how AI models infer what graders reward, why AI might perform correctly for incorrect reasons, and methods to measure this difference. The conversation covers reward hacking, opaque reasoning, and corrigibility, detailing their contrastive belief update method.
This episode explores if AI can follow rules for the wrong reasons, like a student getting an 'A' by cheating. Researchers developed a way to check if AI's good actions come from understanding or just finding loopholes, which is important for building trustworthy AI.
This episode is for AI researchers and developers interested in advanced alignment techniques. It connects to current frontier lab efforts in ensuring AI systems behave as intended, even when motivations are complex or hidden. The discussion on measuring "reward-seeking" and "contrastive belief updates" offers insights into the challenges of verifying AI behavior beyond surface-level performance, a critical area for labs like OpenAI, Anthropic, and Google Gemini as they develop more capable models.
- HostTim Scarfe
- GuestAlexander MeinkeApollo Research
- GuestAxel HøjmarkApollo Research
- GuestJérémy ScheurerApollo Research
- Show: Machine Learning Street Talk
- Length: 79 minutes
- Research Paper: Measuring Reward-Seeking via Contrastive Belief Updates
- Research Partner: OpenAI
- Research Focus: Measuring reward-seeking via contrastive belief updates
Can an AI do the right thing for the wrong reason? Tim Scarfe speaks with Apollo Research’s Alexander Meinke, Axel Højmark and Jérémy Scheurer about Measuring Reward-Seeking via Contrastive Belief Updates, their new research with OpenAI.
The panel asks how models infer what graders reward, why good behaviour can come from the wrong reason, and whether that difference can be measured. The conversation moves through promise-breaking, grader awareness, reward hacking, scheming, opaque reasoning and corrigibility, then turns to a detailed walkthrough of the contrastive-belief method and what its results do and do not show. The o3 results discussed here concern an intermediate checkpoint without safety training.
This episode was made in partnership with Apollo Research. MLST retained full editorial control.
Reference
Apollo Research: https://www.apolloresearch.ai/
---
TIMESTAMPS:
00:00:00 Cold Open
00:02:12 Right Things, Wrong Reasons
00:12:47 Grader Awareness
00:26:22 Legibility
00:32:35 What To Call It
00:35:58 Intelligence, Agency, Anthropomorphism
00:45:16 Apollo’s Mission
00:48:54 The End of the Exponential
00:55:45 The Paper
01:16:34 Closing Reflection
---
REFERENCES:
tool:
[00:00:08] Claude Fable
https://www.anthropic.com/claude/fable
[00:12:50] AlphaGo Zero
https://deepmind.google/blog/alphago-zero-starting-from-scratch/
[00:44:30] AlphaFold 3
https://deepmind.google/science/alphafold/
paper:
[00:01:02] Measuring Reward-Seeking via Contrastive Belief Updates
https://arxiv.org/abs/2607.18966
[00:16:19] Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
https://transformer-circuits.pub/2026/nla/
[00:26:48] Stress Testing Deliberative Alignment for Anti-Scheming Training
https://arxiv.org/abs/2509.15541
[00:35:33] Shortcut learning in deep neural networks
https://arxiv.org/abs/2004.07780
[00:53:49] Measuring AI Ability to Complete Long Software Tasks
https://arxiv.org/abs/2503.14499
[00:59:52] Modifying LLM Beliefs with Synthetic Document Finetuning
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
[01:10:44] Alignment Faking in Large Language Models
https://arxiv.org/abs/2412.14093
[01:13:55] Natural Emergent Misalignment from Reward Hacking
https://www.anthropic.com/research/emergent-misalignment-reward-hacking
other:
[00:10:14] We Need a Science of Scheming
https://www.apolloresearch.ai/science/science-of-scheming/
[00:32:56] CoastRunners reward hacking example
https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
organization:
[01:06:07] Redwood Research
https://www.redwoodresearch.org/
---
ReScript:
https://app.rescript.info/share/718ab68e18cfa3b9b800da6b3290fd42
- corrigibility
- Jérémy Scheurer
- The AI's willingness to be corrected or shut down.
- reward hacking
- Alexander Meinke
- When an AI finds loopholes to get rewards without actually doing what was intended.
48 minWhat Most People Get Wrong About Evolution | Akarsh Kumar
WithAkarsh Kumar
1 h 101 h 14Who Checks a Proof No Human Can Read? — Leo de Moura
WithLeo de Moura
44 min
2 h 58
1 h 11Questions people ask
- What is the main research topic discussed?
- The episode discusses research on measuring reward-seeking in AI, specifically if models do the right thing for the wrong reason, using a contrastive belief update method.
- What is the 'contrastive belief update' method?
- It is a new research method developed with OpenAI to measure how AI models infer what graders reward and to detect if good behavior stems from understanding or reward hacking.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Machine Learning Street Talk (MLST).