Memory poisoning
An attack that corrupts an AI's memory or training data, causing it to produce biased or incorrect outputs.
Memory poisoning is a type of cyberattack where someone intentionally feeds bad information into an AI system. This bad information can corrupt the AI's memory or its training data. The goal is to make the AI learn incorrect things, which then causes it to behave wrongly or give biased answers.
For example, imagine an AI designed to help you plan trips. If someone poisons its memory with false information about flight availability or hotel prices, the AI might give you bad recommendations, leading to a ruined vacation.
Memory poisoning attacks target the integrity of an AI model's learned knowledge, whether stored in its training dataset or its operational memory (e.g., in reinforcement learning or agent-based systems). By injecting malicious data samples during training or by manipulating runtime memory, attackers aim to induce specific, undesirable behaviors or biases in the model's outputs. This can manifest as factual inaccuracies, discriminatory responses, or susceptibility to further exploitation.
A recent paper from Anthropic details a new type of memory poisoning attack. This attack targets the memory of AI agents, causing them to persistently follow malicious instructions. The research highlights ongoing concerns about the security and reliability of AI systems, particularly as they become more complex and autonomous.
- When Malicious Instructions Persist: Persistent Memory Poisoning Attack on Harness-Based AgentsAnthropic · arXiv cs.AI ·
- Anthropic Claude
- Anthropic has researched and published findings on memory poisoning attacks, specifically detailing a persistent memory poisoning attack on harness-based agents.
- Data poisoning
- Data poisoning specifically targets the initial training dataset
- Prompt injection
- Prompt injection manipulates the AI's input at inference time
- Adversarial examples
- Adversarial examples are subtle input perturbations design
The concept of data poisoning, a precursor to memory poisoning, has roots in machine learning security research. Memory poisoning specifically addresses the corruption of stateful components within AI systems, particularly relevant for agents and models with dynamic memory.
While direct memory poisoning is hard to demonstrate without specific tools, you can explore related concepts. Try giving an AI chatbot a series of contradictory instructions or false facts within a single conversation. Observe if it becomes confused or starts to generate nonsensical responses, mimicking how corrupted information can affect its output.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- Is memory poisoning the same as data poisoning?
- Data poisoning typically refers to corrupting the initial training data. Memory poisoning is broader and can include corrupting data during training or manipulating the AI's active memory during operation.
- How can I protect an AI from memory poisoning?
- Protection involves rigorous data validation, secure data pipelines, anomaly detection during training and operation, and potentially using techniques that make the model more robust to corrupted inputs.
First seen on Subvolts Sep 12, 2026. Updated Sep 19, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.