Subvolts take Find LLM agent failures standard tests miss using vector fingerprints and better observability. May 7, 2026 - Developers
- Learn practical techniques for instrumenting LLM agents and uncovering failures missed by standard evaluations.
- Business
- Understand how to improve the reliability and performance of complex AI systems operating in production environments.
This episode offers practical advice for improving LLM agent reliability in production, a key challenge for AI development.
How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
Audio streams from Sam Charrington. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Scott Clark, co-founder of Distributional, joins The TWIML AI Podcast to discuss operating and improving complex LLM systems in production. He introduces a hierarchy of observability and explains how to find agent failures missed by standard evaluations. The episode covers uncovering emergent behaviors, feeding data back into models, and practical instrumentation techniques.
This episode explains how to find hidden problems in AI systems that regular checks miss. By looking at AI behavior patterns, teams can discover unexpected issues and make their AI work better and more reliably.
This episode is for ML engineers and AI practitioners working with LLM agents in production. It addresses the critical challenge of identifying failures that standard evaluation methods overlook, such as "lazy" tool use. Clark's approach, mapping traces into vector fingerprints for clustering, offers a concrete method to surface unknown unknowns. This is highly relevant as frontier labs like Google Gemini and OpenAI focus on making their AI systems more robust and reliable in real-world applications.
- HostSam Charrington
- GuestScott Clarkco-founder and CEO, Distributional
- Guest: Scott Clark, co-founder and CEO of Distributional
- Title: How to Find the Agent Failures Your Evals Miss
- Show: The TWIML AI Podcast
- Length: 53 minutes
- Observability Hierarchy: Telemetry, Monitoring, Post-production/Online Analytics
- Failure Example: Lazy tool-use hallucinations
In this episode, Scott Clark, co-founder and CEO of Distributional, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools.
The complete show notes for this episode can be found at https://twimlai.com/go/767.
- LLM
- Scott Clark
- Large Language Model, a type of AI that understands and generates text.
- telemetry
- Scott Clark
- Data collected automatically about system performance and behavior.
- evals
- Scott Clark
- Evaluations, tests used to measure how well an AI system performs.
- vector fingerprints
- Scott Clark
- Numerical representations of data that allow for grouping similar patterns.
53 min1 h 30Why Jev Is Changing How We Build With AI with Diogo Almeida - #779
WithDiogo Almeida
1 h 081 h 05From Voice Agents to AI Avatars with Alexander Smola - #777
WithAlexander Smola
1 h 05
38 min
45 min
43 minQuestions people ask
- How can teams find LLM agent failures that standard evaluations miss?
- Teams can map traces into vector fingerprints to enable clustering and topic discovery, uncovering emergent behaviors and unknown unknowns.
- What is Scott Clark's hierarchy of observability?
- It includes telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Sam Charrington.