Use clustered standard errors for non-independent eval questions
Account for related questions in evals by clustering standard errors on the unit of randomization to avoid underestimating uncertainty.
When the inclusion of questions is non-independent, we recommend clustering standard errors on the unit of randomization.
Popular evals often have groups of closely related questions (e.g., multiple questions about the same text passage). Naively applying the Central Limit Theorem can underestimate the standard error, potentially leading to incorrect conclusions about model capabilities.
For reading comprehension evals, cluster standard errors on the passage of text.
If an AI model evaluation contains groups of related questions (e.g., multiple questions on the same text), use clustered standard errors. This method, adapted from social sciences, correctly accounts for the non-independence of questions and prevents underestimating uncertainty.
Ignoring question clustering can lead to underestimating the standard error, potentially misleading analysts into drawing incorrect conclusions from the data. Clustered standard errors provide a more accurate measure of uncertainty when questions are not independent.
- A statistical approach to model evaluations ↗Anthropic · Anthropic Research · Nov 19, 2024 · paraphrased
First seen on Subvolts Nov 19, 2024. Updated Sep 11, 2026. Spotted a problem? Tell us.
- A statistical approach to model evaluationsAnthropic · Anthropic Research · Nov 19, 2024
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- Troubleshooting fixUnderestimated standard error in non-independent eval questionsUse clustered standard errors when eval questions are related to avoid underestimating uncertainty and drawing incorrect conclusions. · 5 min
- Eval methodEvaluate LLM forecasting performance with structured effectsMeasure how much structured future information improves LLM time series forecasting accuracy. · 30 min
- TemplatePrompt Library for Life Sciences TasksA collection of pre-written prompts designed for optimal results on common life sciences tasks. · 2 min
- HeuristicAI response evaluation criteria in mental healthUse weighted criteria to assess AI responses, rewarding beneficial behaviors and penalizing harmful ones. · 20 min
- ChecklistPre-computation Checklist for AI in Life SciencesItems to consider before using AI for complex life sciences tasks. · 5 min
- ChecklistLLM knowledge profiling: encoding and recall assessmentChecklist for assessing LLM factuality by measuring encoding and recall. · 30 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- Anthropic (Anthropic Research). Subvolts collected, normalized and credited it; the link above opens the original.