Distillation attacks
An attack where a smaller AI model learns to copy a larger, more capable AI model, often by studying its outputs.
Distillation attacks happen when someone tries to steal the knowledge or capabilities of a powerful AI model.
They do this by training a smaller, cheaper AI model to imitate the larger one. The smaller model learns by seeing the outputs of the bigger model, or sometimes by trying to understand its internal workings. The goal is to create a less expensive AI that performs almost as well as the original.
For example, imagine a large AI that can write complex code. An attacker could use distillation to train a smaller AI to write similar code by feeding it many examples of the large AI's code outputs.
Distillation attacks, also known as model extraction or knowledge distillation attacks, involve training a smaller, student model to mimic the behavior and performance of a larger, teacher model. This is typically achieved by querying the teacher model and using its outputs as training data for the student. The attacker aims to replicate the teacher's functionality at a reduced computational cost or to bypass licensing restrictions.
These attacks can be particularly concerning if the teacher model contains proprietary information or has been trained on sensitive data. The success of distillation depends on the attacker's ability to generate informative queries and the teacher model's susceptibility to revealing its learned patterns through its responses.
Anthropic has recently highlighted its efforts to defend against distillation attacks, indicating these attacks are a current concern for leading AI labs. These attacks aim to replicate the capabilities of advanced models, posing a threat to proprietary AI research and development.
- CoreWeave Crash, AI Trade Cracks, Anthropic IPO Red Flags, Bill Gates DoomerAnthropic · All-In Podcast ·
- Strengthening our defenses against distillation attacksAnthropic · Anthropic Research ·
- Anthropic Says It Blocked Bioweapons Efforts and Detected Chinese Distillation AttacksAnthropic · The Information ·
- Why Claude switched models in your conversation with Fable 5 or Fable 5.1Anthropic · Claude Help Center ·
- It's Hard Out There For a LobsterAnthropic · AI Inside ·
- Anthropic Claude
- Anthropic has publicly discussed strengthening its defenses against distillation attacks and has detected such attempts, including those from China.
- Model extraction
- This is a broader term that includes distillation attacks, but
- Adversarial attacks
- These attacks aim to fool an AI into making incorrect predi
- Data poisoning
- This involves corrupting the training data of an AI model, rathe
The concept of knowledge distillation, where a smaller model learns from a larger one, was popularized by Geoffrey Hinton and others in a 2015 paper. The term 'distillation attack' refers to the malicious application of this technique. Source ↗
While directly performing a distillation attack is complex, you can observe the principle by comparing the outputs of different model tiers. For instance, try asking ChatGPT 3.5 to perform a complex creative writing task, then ask GPT-4 to do the same. Notice the differences in quality and nuance, which represent the 'knowledge' a distillation attack would try to capture.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- Why is it called a 'distillation' attack?
- It's called distillation because, like distilling a liquid to extract its essence, the attack aims to extract the core knowledge or capabilities from a large AI model into a smaller one.
- Is distillation always an attack?
- No, knowledge distillation is also a legitimate technique used in machine learning to compress large models into smaller, more efficient ones for deployment. It becomes an attack when done without permission to steal proprietary AI capabilities.
First seen on Subvolts Sep 3, 2026. Updated Oct 17, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.