Subvolts take Learn how to assess AI risks beyond benchmarks and manage real-world failures. Feb 13, 2026 - Developers
- Learn practical methods for assessing and managing AI systems to prevent real-world failures.
- Business
- Understand the limitations of AI benchmarks and how to better evaluate AI risks in deployment.
This episode provides a practical discussion on AI safety and evaluation with an expert in AI incidents.
AI incidents, audits, and the limits of benchmarks
Audio streams from Daniel Whitenack and Chris Benson. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
This Practical AI episode features Sean McGregor discussing AI safety, verification, and evaluation. Hosts Daniel Whitenack and Chris Benson explore why benchmarks fall short, the risks revealed by red-teaming at DEFCON, and practical methods for assessing and managing AI systems. The conversation focuses on moving AI from research to real-world deployment and addressing potential failures.
This episode discusses how AI systems can fail in the real world, even if they seem to work well in tests. It explores why standard tests are not enough and how to better check and manage AI to prevent problems.
This episode offers insights into the practical challenges of AI safety and evaluation as systems move into production. It's for AI practitioners, researchers, and policymakers concerned with real-world AI risks. The discussion on benchmarks and incident reporting is relevant as major labs like Google Gemini, OpenAI, and Anthropic grapple with ensuring their advanced models are safe and reliable before widespread deployment.
- HostDaniel Whitenack
- HostChris Benson
- GuestSean McGregorco-founder, AI Verification & Evaluation Research Institute
- GuestSean McGregorfounder, AI Incident Database
- Guest: Sean McGregor
- Show: Practical AI
- Length: 43 minutes
- Guest Affiliation 1: AI Verification & Evaluation Research Institute
- Guest Affiliation 2: AI Incident Database
Chapters from the publisher ↗ · Transcript ↗ Key moments and transcript search come with a later piece.
AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean McGregor, co-founder of the AI Verification & Evaluation Research Institute and also the founder of the AI Incident Database, joins Chris and Dan to discuss AI safety, verification, evaluation, and auditing. They explore why benchmarks often fall short, what red-teaming at DEFCON reveals about machine learning risks, and how organizations can better assess and manage AI systems in practice.
Featuring:
- Sean McGregor– LinkedIn
- Chris Benson – Website, LinkedIn, Bluesky, GitHub, X
- Daniel Whitenack – Website, GitHub, X
Links:
- AI Verification & Evaluation Research Institute
- AI Incident Database
- 38th convening of IAAI
- BenchRisk
- State of Global AI Incident Reporting
Upcoming Events:
- Register for upcoming webinars here!
- red-teaming
- Sean McGregor
- testing AI systems by trying to make them fail or behave badly
- benchmarks
- Daniel Whitenack, Chris Benson, Sean McGregor
- standardized tests used to measure AI performance
44 minNarrative intelligence and the human advantage
WithReed Frerichs
47 minOpen models and the future of Physical AI with NVIDIA
WithMing-Yu Liu
49 minFrom AGENTS.md to Enterprise Deployment
WithNick Kuhn
Questions people ask
- What is the AI Incident Database?
- It is a resource founded by Sean McGregor that tracks and documents incidents involving artificial intelligence systems.
- Why are AI benchmarks often insufficient?
- The episode explores why benchmarks frequently fall short in evaluating AI systems for real-world deployment and safety.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Daniel Whitenack and Chris Benson.