Report standard error of the mean (SEM) alongside eval scores
Quantify uncertainty in eval scores by reporting the SEM, derived from the Central Limit Theorem, to understand score reliability.
Encourage researchers to report the SEM, derived from the Central Limit Theorem, alongside each calculated eval score.
The SEM indicates the standard deviation of the sampling distribution of the mean. Reporting it allows for quantifying the difference in theoretical means between two models and calculating confidence intervals.
A 95% confidence interval can be calculated from the SEM by adding and subtracting 1.96 × SEM from the mean score.
When reporting AI model evaluation scores, include the Standard Error of the Mean (SEM) alongside the average score. This statistical measure, derived from the Central Limit Theorem, helps quantify the uncertainty and reliability of the score.
Under the Central Limit Theorem, the mean values of several random samples taken from the same underlying distribution tend to follow a normal distribution. The SEM is the standard deviation of this distribution, indicating how much the sample mean is likely to vary.
- A statistical approach to model evaluations ↗Anthropic · Anthropic Research · Nov 19, 2024 · paraphrased
First seen on Subvolts Nov 19, 2024. Updated Sep 11, 2026. Spotted a problem? Tell us.
- A statistical approach to model evaluationsAnthropic · Anthropic Research · Nov 19, 2024
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- ChecklistScientific Reporting for AI Model EvaluationsChecklist for scientifically informative reporting of AI model evaluation results. · 10 min
- Troubleshooting fixImprove LLM Saudi dialect competence by penalizing errorsPenalize LLMs for errors like ambiguous framing and hallucination when evaluating Saudi dialect competence. · 20 min
- HeuristicTicket wording matters more than claims for LLM diagnosisHow a network fault ticket is written significantly impacts LLM diagnostic performance, more than its stated claims. · 5 min
- ToolScrewtape adversarial transcript editing pipeline for AI monitorsPipeline converts real agent transcripts into attacks that evade action-only monitors. · 120 min
- Eval methodEvaluate LLM forecasting performance with structured effectsMeasure how much structured future information improves LLM time series forecasting accuracy. · 30 min
- ChecklistAudit LLM evaluation for citation artifacts and biasCheck for presentation cues, citation issues, and quality gaps in LLM-as-a-judge evaluations. · 30 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- Anthropic (Anthropic Research). Subvolts collected, normalized and credited it; the link above opens the original.