KL divergence
Also written: Kullback-Leibler divergence
A way to measure how different two sets of outcomes are, often used to compare AI models.
Imagine you have a recipe for baking cookies, and you try a new variation. KL divergence is like a score that tells you how much the new cookies' taste and texture differ from the original recipe's expected outcome.
In AI, we use this idea to compare how an AI model behaves or predicts things compared to another version or a desired behavior. For example, if an AI is supposed to generate helpful answers, we can use KL divergence to see how much its actual answers differ from what we consider perfectly helpful.
Kullback-Leibler (KL) divergence quantifies how one probability distribution, P, diverges from a second, reference probability distribution, Q. It is defined as the expected value of the logarithmic difference between the probabilities of P and Q. KL(P || Q) = E_x [log(P(x) / Q(x))]. It is a non-symmetric measure, meaning KL(P || Q) is not generally equal to KL(Q || P). In machine learning, it's often used as a regularization term to keep a model's output distribution close to a target distribution, or to measure the difference between a learned distribution and a prior.
KL divergence is appearing in discussions about training AI assistants to align with human preferences, as seen in Anthropic's RLHF work. It's also relevant for evaluating AI progress and ensuring models don't overfit to specific metrics, as explored by OpenAI.
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human FeedbackAnthropic · Anthropic Research ·
- Measuring Goodhart’s lawOpenAI · OpenAI News ·
- Anthropic Claude
- Anthropic uses KL divergence in reinforcement learning from human feedback to guide AI behavior towards desired outcomes.
- OpenAI ChatGPT
- OpenAI has explored KL divergence in the context of measuring AI progress and avoiding metric gaming.
- Jensen-Shannon divergence
- A symmetric and smoothed version of KL divergence, of
- Total Variation Distance
- A measure of the maximum difference between probabilit
- Cross-Entropy
- Closely related to KL divergence, cross-entropy is often used as
The Kullback-Leibler divergence was introduced by Solomon Kullback and Richard Leibler in their 1951 paper on information theory. Source ↗
Ask an AI chatbot to write a short story about a cat. Then, ask it to write another story about the same cat, but this time make it 'more adventurous.' Compare the two stories. While not a direct KL divergence calculation, you're observing how the AI's output distribution (the stories) changes based on a modified prompt, similar to how KL divergence measures differences between distributions.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- HeuristicEstimate true objective and KL divergence with best-of-n samplingEfficiently estimate true objective and KL divergence using samples from the original distribution. · 15 min
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
Questions people ask
- Is KL divergence always positive?
- KL divergence is always non-negative. It is zero if and only if the two probability distributions are identical.
- Why is KL divergence not symmetric?
- The asymmetry reflects that KL(P || Q) measures the information lost when Q is used to approximate P, which is different from the information lost when P is used to approximate Q.
First seen on Subvolts Apr 13, 2022. Updated Sep 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.