Underestimated standard error in non-independent eval questions
Use clustered standard errors when eval questions are related to avoid underestimating uncertainty and drawing incorrect conclusions.
Use clustered standard errors, clustering on the unit of randomization (e.g., the text passage), to accurately account for the non-independence of questions and provide a more reliable measure of uncertainty.
Underestimated standard error in AI model evaluation scores.
Naively applying statistical methods (like the Central Limit Theorem) to evaluation datasets where questions are not independent, such as multiple questions about the same text passage.
If your AI model evaluations have groups of related questions, a naive application of statistics can lead to underestimated standard errors. The fix is to use clustered standard errors, which correctly account for this non-independence and prevent misleading conclusions.
Clustered standard errors properly adjust for the fact that questions related to the same unit (like a passage) are not independent, leading to a more accurate assessment of statistical uncertainty.
- A statistical approach to model evaluations ↗Anthropic · Anthropic Research · Nov 19, 2024 · paraphrased
First seen on Subvolts Nov 19, 2024. Updated Sep 11, 2026. Spotted a problem? Tell us.
- A statistical approach to model evaluationsAnthropic · Anthropic Research · Nov 19, 2024
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- HeuristicUse clustered standard errors for non-independent eval questionsAccount for related questions in evals by clustering standard errors on the unit of randomization to avoid underestimating uncertainty. · 5 min
- Eval methodEvaluate LLM forecasting performance with structured effectsMeasure how much structured future information improves LLM time series forecasting accuracy. · 30 min
- TemplatePrompt Library for Life Sciences TasksA collection of pre-written prompts designed for optimal results on common life sciences tasks. · 2 min
- HeuristicAI response evaluation criteria in mental healthUse weighted criteria to assess AI responses, rewarding beneficial behaviors and penalizing harmful ones. · 20 min
- ChecklistPre-computation Checklist for AI in Life SciencesItems to consider before using AI for complex life sciences tasks. · 5 min
- ChecklistLLM knowledge profiling: encoding and recall assessmentChecklist for assessing LLM factuality by measuring encoding and recall. · 30 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- Anthropic (Anthropic Research). Subvolts collected, normalized and credited it; the link above opens the original.