Top news
Investing in the public benefit OfficialAnthropic NewsOct 18Popular Musk Pledges SpaceX Shares to ‘Every Kid in America’ WebBloomberg TechnologyOct 19Popular Websites are closing their doors to AI agents WebThe Verge AIOct 17 This is the new ChatGPT Voice, powered by GPT Live OfficialOpenAI on YouTubeOct 18 How AI giants will reshape global philanthropy WebAxios AIOct 19 Gemini 4 Argon Really Surprised Me WebTwo Minute PapersOct 19 Google Gemini 4 agents just beat Minecraft with no human intervention WebEngadget AIOct 19 Who could it be? 🧐 OfficialGoogle on YouTubeOct 18 Anthropic gives outside AI watchdog “employee-like” access WebArs Technica AIOct 18 What are self-replicating ‘AI worms’, and why is Anthropic worried about them? WebThe Guardian AIOct 18
GlossaryTrainingNew to AI

Distillation

Also written: Distillation notebook, Distillation campaigns, Distilling, Adversarial distillation, Distilling outputs, knowledge distillation, Distilled reasoning model

Training a smaller AI model to copy the performance of a larger, more capable AI model.

In plain terms

Distillation is like a student AI learning from a master AI. The master AI is very smart and can do many things, but it's also big and slow. The student AI is smaller and faster, but it doesn't know as much.

To train the student AI, we show it the answers or actions of the master AI. The student AI tries to copy these answers. Over time, it gets good at doing what the master AI does, becoming a more efficient version. For example, a large language model could be distilled into a smaller model that can run on a phone, offering many of the same capabilities but with less power.

This process makes AI more accessible. Smaller models can be used on devices with less computing power, like smartphones or laptops, and they are cheaper to run. It's a way to get the benefits of powerful AI without needing supercomputers.

The expert version

Model distillation is a training technique where a smaller, "student" model is trained to replicate the behavior or outputs of a larger, pre-trained "teacher" model. The student model learns to mimic the teacher's predictions, often by minimizing a loss function that includes the difference between the student's outputs and the teacher's "soft targets" (probability distributions over classes) or direct predictions.

This method aims to transfer the knowledge and generalization capabilities of a large, complex model to a more compact and computationally efficient one. Tradeoffs include potential loss of performance compared to the teacher model, but the benefits are reduced inference latency, lower memory footprint, and decreased computational cost, making deployment on resource-constrained devices feasible.

Why it matters now

Distillation is a growing concern for frontier AI labs as bad actors attempt to replicate their advanced models. Recent reports highlight coordinated campaigns aimed at extracting proprietary AI capabilities, forcing companies to develop defenses against these 'thinking' extraction methods. This has become a new battleground for protecting AI intellectual property.

Where it shows up 29 items, newest first
How each company uses it
Anthropic Claude
Anthropic has detailed multiple large-scale distillation campaigns targeting Claude's advanced features from various AI firms.
OpenAI ChatGPT
OpenAI has accused a company of running a distillation campaign and is working to disrupt coordinated efforts to extract AI model behavior.
Related terms

Model compressionKnowledge distillationTeacher-student modelsModel quantizationFine-tuningTransfer learning

Often confused with
Fine-tuning
Fine-tuning adapts a pre-trained model to a specific task using a s
Model quantization
Quantization reduces the precision of model weights to decre
Pruning
Pruning removes redundant weights or neurons from a model to make it sm
Origin

The concept of knowledge distillation was popularized by Geoffrey Hinton and his colleagues in a 2015 paper, though earlier work explored similar ideas of model compression and knowledge transfer. Source ↗

Try this about 5 minutes

You can't directly perform distillation in 5 minutes with current frontier AI products. However, you can observe its effects by comparing the performance and speed of a large model (like ChatGPT or Claude) to a smaller, specialized model if one is available. Notice how the larger model might be more versatile but slower, while a smaller model could be faster for specific tasks.

Try it: Playbook entries that use thisPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Questions people ask

Why would I want to distill a model?
Distillation creates smaller, faster, and cheaper AI models that can run on less powerful hardware, like mobile phones, while retaining much of the capability of the original large model.
Does the smaller model perform exactly like the bigger one?
Not exactly. The smaller model aims to mimic the bigger one, but some performance or nuance might be lost in the process. The goal is to get 'good enough' performance with significant efficiency gains.

First seen on Subvolts Jun 20, 2024. Updated Oct 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.