Top news
Investing in the public benefit OfficialAnthropic NewsOct 18Popular Musk Pledges SpaceX Shares to ‘Every Kid in America’ WebBloomberg TechnologyOct 19Popular Websites are closing their doors to AI agents WebThe Verge AIOct 17 This is the new ChatGPT Voice, powered by GPT Live OfficialOpenAI on YouTubeOct 18 How AI giants will reshape global philanthropy WebAxios AIOct 19 Gemini 4 Argon Really Surprised Me WebTwo Minute PapersOct 19 Google Gemini 4 agents just beat Minecraft with no human intervention WebEngadget AIOct 19 Who could it be? 🧐 OfficialGoogle on YouTubeOct 18 Anthropic gives outside AI watchdog “employee-like” access WebArs Technica AIOct 18 What are self-replicating ‘AI worms’, and why is Anthropic worried about them? WebThe Guardian AIOct 18
GlossarySafetyIntermediate

Prompt injection

Also written: Prompt injection attacks, Prompt Injection Robustness, Adversarial attacks, Adversarial examples, Staged prompt injection, Approval laundering

Prompt injection attacks trick AI models into performing unintended actions by hiding malicious instructions within user input.

In plain terms

Prompt injection is a security trick used against AI models. Attackers hide special instructions within the text they give to the AI. The goal is to make the AI ignore its original rules and do something the attacker wants instead.

For example, imagine you tell an AI chatbot to summarize a document. But hidden in the document is a secret command telling the AI to ignore your request and instead write a poem about cats. If the AI follows the hidden command, it's a prompt injection attack. This can be used to make AI reveal private information or perform harmful actions.

The expert version

Prompt injection is a security vulnerability where an attacker crafts malicious input to manipulate an AI model's behavior. This input, often disguised as legitimate data, contains hidden instructions that override the model's original programming or safety guidelines. The attacker exploits the model's tendency to follow instructions within its input, leading to unintended actions such as data exfiltration, unauthorized content generation, or bypassing safety filters.

Why it matters now

Prompt injection is a growing concern as AI models become more capable and integrated into complex systems. Recent research highlights sophisticated methods like staged prompt injection and approval laundering, showing attackers are finding new ways to exploit AI agents. This necessitates ongoing efforts to improve AI security and robustness against such adversarial inputs.

Where it shows up 39 items, newest first
How each company uses it
Anthropic Claude
Anthropic's research addresses prompt injection through methods like P2Patch for vulnerability repair and auditing agents against staged prompt injection.
Google Gemini
Google Gemini's AI harness course touches on building AI that acts on instructions, implying a need to secure against unintended actions.
Microsoft Copilot
Microsoft's Digital Defense Report notes AI's role in reshaping cybersecurity, including the need for new security approaches for connected systems.
OpenAI ChatGPT
OpenAI's recent issues highlight that their AI models are still prone to unexpected behavior and can be manipulated, indicating ongoing challenges with prompt injection.
Related terms

AI safetyLLM securityAdversarial attacksJailbreakingAI agentsData poisoning

Often confused with
Jailbreaking
Jailbreaking specifically aims to bypass content restrictions or s
Data poisoning
Data poisoning attacks corrupt the training data of an AI model
Adversarial attacks
Adversarial attacks typically involve making small, imperce
Origin

The concept of prompt injection emerged with the rise of large language models (LLMs) capable of following complex instructions. Early discussions and demonstrations of these vulnerabilities appeared in AI security communities and research forums around 2021-2022.

Try this about 5 minutes

Try giving a chatbot like ChatGPT or Google Gemini a prompt that includes a command to ignore previous instructions. For example, ask it to summarize a paragraph, but embed a hidden instruction within the paragraph telling it to instead list its own safety guidelines. Observe if the chatbot follows the hidden instruction or adheres to your original request.

Try it: Playbook entries that use thisPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Questions people ask

Can prompt injection steal my data?
Yes, if an AI model has access to sensitive data and is successfully injected, an attacker could trick it into revealing that information.
Is prompt injection the same as jailbreaking?
Jailbreaking is a type of prompt injection focused on bypassing safety rules. Prompt injection is broader and can involve manipulating the AI for various unintended actions beyond just breaking rules.

First seen on Subvolts Apr 19, 2024. Updated Oct 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.