Using Screwtape to find confirmed failures of AI monitors
Demonstrates how a red-teamer can find confirmed failures of an AI monitor without developer infrastructure access.
Developers can use these attacks to fix monitor blind spots, and evaluators can use them as counterexamples to safety.
Anthropic researchers developed Screwtape, a pipeline for adversarial transcript editing to identify vulnerabilities in AI safety monitors.
They applied Screwtape to 94 Claude Code and Codex transcripts, optimizing against their own monitors. Across 252 runs, it produced 35 attacks that passed the verifier and evaded these monitors. Six of these, confirmed valid by human review, also evaded a held-out monitor provided by a frontier developer.
Confirmed failures of the developer's monitor were found by a red-teamer without access to the developer's infrastructure. More attacks are expected when the pipeline runs inside the developer's infrastructure with frontier attacker models.
A case study on using the Screwtape pipeline to identify vulnerabilities in AI safety monitors. The method successfully generated attacks that evaded monitors, demonstrating its effectiveness for red-teaming and improving AI safety.
The pipeline automates the generation of adversarial examples by using LLMs to edit transcripts, allowing for systematic testing and discovery of monitor weaknesses.
- Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming ↗Anthropic · arXiv cs.AI · Oct 14, 2026 · paraphrased
First seen on Subvolts Oct 14, 2026. Updated Oct 19, 2026. Spotted a problem? Tell us.
- Screwtape: Adversarial Transcript Editing for Monitor Red-TeamingAnthropic · arXiv cs.AI · Oct 14, 2026
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- Eval methodEvaluate Constitutional Classifier robustness against jailbreaksTest Constitutional Classifiers by attempting jailbreaks with red teaming and synthetic prompts. · 30 min
- HeuristicAuxiliary environment signals improve LLM authorization accuracyProviding LLMs with auxiliary environment signals can substantially improve their accuracy in authorization decisions. · 30 min
- ToolCData Connect AI for enterprise system accessProvides AI assistants live, permission-aware access to hundreds of enterprise systems. · 15 min
- HeuristicWeb browsing amplifies AI prompt injection risksThe vast attack surface and diverse actions of browser agents significantly amplify prompt injection risks. · 2 min
- ChecklistAI System Red-Teaming for Election MisuseChecklist for testing AI systems for potential misuse during elections. · 60 min
- ToolScrewtape adversarial transcript editing pipeline for AI monitorsPipeline converts real agent transcripts into attacks that evade action-only monitors. · 120 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with Claude in the sources.
- Where did this come from?
- Anthropic (arXiv cs.AI). Subvolts collected, normalized and credited it; the link above opens the original.