Subvolts take Understand AI safety challenges and UK efforts to evaluate frontier models. Mar 1, 2026 - Developers
- Learn about advanced AI evaluation techniques and safety research priorities from a national security perspective.
- Business
- Understand the risks and safety measures being developed for powerful AI systems that could impact your industry.
- Everyday users
- Gain insight into the efforts to make AI safer and more reliable before widespread deployment of advanced systems.
A solid interview with a key figure from a national AI safety institute, discussing critical evaluation and research efforts.
Situational Awareness in Government, with UK AISI Chief Scientist Geoffrey Irving
Audio streams from Erik Torenberg, Nathan Labenz. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Geoffrey Irving, Chief Scientist at the UK AI Security Institute, discusses the fragile theoretical understanding of machine learning despite advanced models. He details the AISI's work on evaluating frontier AI, red teaming, and threat modeling for biosecurity, cybersecurity, and loss-of-control risks. The conversation covers reward hacking, eval awareness, and the limitations of current safety techniques, as well as AISI's funding for foundational AI safety research.
This episode explains why we don't fully understand how advanced AI works, even as it gets very good at tasks. It covers how experts test these AIs for safety and what research is needed to make them more reliable for everyone.
This episode is for AI safety researchers, policymakers, and anyone concerned about the risks of advanced AI. It connects directly to ongoing efforts by frontier labs like OpenAI, Google Gemini, and Anthropic to develop and deploy powerful AI systems responsibly. Irving's insights into evaluation, red teaming, and foundational research highlight the critical need for robust safety guarantees as AI capabilities accelerate, a core challenge for all major AI developers.
- HostNathan Labenz
- GuestGeoffrey IrvingChief Scientist, UK AI Security Institute
- Guest: Geoffrey Irving, Chief Scientist at the UK AI Security Institute
- Show: The Cognitive Revolution
- Length: 139 minutes
- AISI Work: Frontier model evaluations, red teaming, threat modeling (biosecurity, cybersecurity, loss-of-control)
- Discussion Points: Reward hacking, eval awareness, limitations of current safety techniques
Geoffrey Irving, Chief Scientist at the UK AI Security Institute, explains why our theoretical understanding of machine learning remains fragile even as models surpass experts on critical security tasks. He details AISI’s work on frontier model evaluations, red teaming, and threat modeling across biosecurity, cybersecurity, and loss-of-control risks. The conversation explores reward hacking, eval awareness, and why current safety techniques may struggle to deliver high reliability. Listeners will also hear how AISI is funding foundational research to build stronger guarantees for AI safety.
Nathan uses Granola to uncover blind spots in conversations and AI research. Try it at granola.ai/tcr with code TCR — and if you’re already using it, test his blind spot recipe here: https://bit.ly/granolablindspot
Sponsors:
Serval:
Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week 4 at https://serval.com/cognitive
Claude:
Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr
Tasklet:
Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai
CHAPTERS:
(00:00) About the Episode
(04:09) From physics to ML
(08:52) AGI uncertainty and threats (Part 1)
(18:08) Sponsors: Serval | Claude
(21:29) AGI uncertainty and threats (Part 2)
(27:35) Control, autonomy, alignment (Part 1)
(34:02) Sponsor: Tasklet
(35:14) Control, autonomy, alignment (Part 2)
(38:44) Inside the UK AC
(51:02) Evaluations and jailbreaking
(01:01:17) Emerging capabilities and misuse
(01:14:20) Agents and reward hacking
(01:26:09) Theoretical alignment agenda
(01:38:39) Debate and formal methods
(01:51:19) Limits of formalization
(02:02:27) Future risks and governance
(02:16:23) Episode Outro
(02:18:58) Outro
PRODUCED BY:
SOCIAL LINKS:
Website: https://www.cognitiverevolution.ai
Twitter (Podcast): https://x.com/cogrev_podcast
Twitter (Nathan): https://x.com/labenz
LinkedIn: https://linkedin.com/in/nathanlabenz/
- red teaming
- Geoffrey Irving
- Testing AI systems by trying to make them fail or behave badly, like a security team testing a system.
- reward hacking
- Geoffrey Irving
- When an AI finds a shortcut to get a reward in its training without actually doing the intended task well.
- eval awareness
- Geoffrey Irving
- The AI understanding how it is being evaluated and potentially manipulating the evaluation process.
1 h 27AI:AM: Where's the Steering Wheel? My Agent Set the Password, Hidden Valence, Proofs Find Linux Bugs
WithCameron Berg, Everett Hildenbrandt, Grigore Rosu, Phil Maher
2 h 192 h 021 h 28AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?
WithThomas Sohmers, swyx, Evan Miyazono, Edward Hu
Same guests, other shows
1 h 45AI in the AM — Week 2 Highlights (June 2026)
WithGeoffrey Irving, Daniel Murfet, Rahul Sonwalkar, Shlok Khemani
38 minSpeaking of intelligence
WithGeoffrey Irving, Chris Dyer, Angeliki Lazaridou, Lisa-Anne Hendriks
1 h 01
48 min
Questions people ask
- What is the UK AI Security Institute (AISI) doing?
- AISI evaluates frontier AI models, conducts red teaming, and models threats across biosecurity, cybersecurity, and loss-of-control risks. They also fund foundational research for AI safety.
- What are some challenges in AI safety discussed?
- The episode covers reward hacking, eval awareness, and why current safety techniques may struggle to ensure high reliability for advanced AI systems.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Erik Torenberg, Nathan Labenz.