Measure run-to-run instability in AI model outputs
Evaluate AI model consistency by measuring theme churn and volume disagreement across multiple runs.
Introduce a repeat-run evaluation framework that aligns semantically equivalent categories and focuses on two operating metrics: theme churn, the normalized change in the returned category set, and volume disagreement, the change in counts for categories that persist.
Lower theme churn and volume disagreement indicate greater stability.
Evaluate three recurring customer-feedback tasks across eight frontier models, corpus sizes from 100 to 5,000 records, multiple prompts, and three execution designs.
This evaluation method measures the consistency of AI model outputs across multiple runs. It uses theme churn to track changes in categories and volume disagreement for changes in category counts, helping to identify and reduce instability in AI-driven analysis.
By quantifying changes in category sets and counts, this framework provides objective metrics for AI model stability, crucial for reliable automation over unstructured data.
- Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis ↗arXiv · arXiv cs.AI · Oct 6, 2026 · paraphrased
First seen on Subvolts Oct 6, 2026. Updated Oct 8, 2026. Spotted a problem? Tell us.
- Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback AnalysisAnthropic · arXiv cs.AI · Oct 6, 2026
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- HeuristicUse taxonomy-grounded agents for stable AI analysisEmploy taxonomy-grounded agents to significantly reduce run-to-run instability in AI model outputs. · 30 min
- ChecklistCheck AI model safety evidence acquisition policiesEvaluate how AI models seek safety information before acting using a controlled benchmark. · 120 min
- ChecklistAudit LLM evaluation for citation artifacts and biasCheck for presentation cues, citation issues, and quality gaps in LLM-as-a-judge evaluations. · 30 min
- Eval methodCheck LLM state policy recall for cross-jurisdiction substitutionVerify if an LLM provides another state's policy value when asked about a specific state. · 60 min
- Eval methodMethod for evaluating potential discriminatory impact of LMsUse an LM to generate prompts across scenarios, varying demographics to find discrimination. · 60 min
- HeuristicCombine statistical covariates with foundation model embeddingsSynergistic value from fusing structured statistical data with latent foundation model representations. · 5 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- arXiv (arXiv cs.AI). Subvolts collected, normalized and credited it; the link above opens the original.