Subvolts take AI agents can code hacking tools fast; securing them is harder than you think. Oct 4, 2025 - Developers
- Learn about the unique security risks AI agents introduce and how to build more secure systems.
- Business
- Understand the evolving threat landscape as AI agents interact with sensitive business data and systems.
Features a former DeepMind researcher discussing critical AI security issues, offering valuable insights into agent vulnerabilities.
AI Agents Can Code 10,000 Lines of Hacking Tools In Seconds - Dr. Ilia Shumailov (ex-GDM)
Audio streams from Machine Learning Street Talk (MLST). Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
This Machine Learning Street Talk episode features Dr. Ilia Shumailov discussing the security challenges of AI agents. Shumailov, formerly of DeepMind, explains why securing AI is complex, especially as agents can generate hacking tools rapidly and interact with sensitive systems 24/7, unlike human employees. The discussion highlights the inadequacy of traditional security measures for these advanced AI capabilities.
Imagine AI agents managing your life, like emails and money. They are super fast and can create hacking tools in seconds. This episode explains why current security methods are not enough to protect us from these powerful AI tools.
This episode is for AI security professionals and developers building AI agents. Dr. Shumailov offers insights into the unique vulnerabilities of AI systems, particularly their ability to generate malicious code and operate continuously. This directly relates to frontier labs like OpenAI, Google Gemini, and Anthropic as they develop increasingly autonomous agents that will interact with user data and systems, necessitating robust security frameworks.
- HostTim Scarfe
- GuestIlia ShumailovFormer DeepMind AI Security Researcher, DeepMind
- Guest: Dr. Ilia Shumailov
- Former Affiliation: DeepMind
- Episode Title: AI Agents Can Code 10,000 Lines of Hacking Tools In Seconds
- Show: Machine Learning Street Talk
- Length: 61 minutes
- Concrete Claim: AI agents can generate sophisticated hacking tools in seconds.
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and he's here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—Check out notebooklm for your research project, it's really powerfulhttps://notebooklm.google.com/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
We're racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply won't work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
https://iliaishacked.github.io/
https://sequrity.ai/
TRANSCRIPT:
https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
https://arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design.
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
https://arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
https://www.anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
https://arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
https://arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
https://openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
https://www.cisa.gov/news-events/news/apa
48 minWhat Most People Get Wrong About Evolution | Akarsh Kumar
WithAkarsh Kumar
1 h 101 h 14Who Checks a Proof No Human Can Read? — Leo de Moura
WithLeo de Moura
44 minSame guests, other shows
49 minStealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov
WithIlia Shumailov, Alexander Panfilov
44 minThe AI industry is at a major crossroads
WithKanjun Qiu
44 minThe impact of AI on the workforce: A state-level case study
WithChelsea Linder
Questions people ask
- What is the main security concern with AI agents?
- AI agents can generate sophisticated hacking tools in seconds and interact with sensitive data 24/7, posing a greater risk than human employees.
- Why are traditional security measures insufficient for AI agents?
- Traditional security is designed for human behavior and systems. AI agents' speed, constant operation, and ability to touch all system endpoints simultaneously create new vulnerabilities.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Machine Learning Street Talk (MLST).