Audit LLM evaluation for citation artifacts and bias
Check for presentation cues, citation issues, and quality gaps in LLM-as-a-judge evaluations.
Before or after
before
Context
Evaluating LLM responses using an LLM-as-a-judge setup.
What it does and when to use it
Use this checklist to audit LLM-as-a-judge evaluations. It guides you to look for presentation cue biases, content equivalence, citation issues, correctness conflicts, and quality gaps to ensure a more reliable assessment.
Why it works
This structured approach helps uncover subtle biases and measurement resolution issues that can arise when LLMs judge other LLMs, leading to more accurate evaluations.
Sources and credit
- Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation ↗arXiv · arXiv cs.AI · Sep 10, 2026 · paraphrased
First seen on Subvolts Sep 10, 2026. Updated Sep 15, 2026. Spotted a problem? Tell us.
Where it shows up
- Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge EvaluationAnthropic · arXiv cs.AI · Sep 10, 2026
Terms used
Related in the PlaybookPlaybook →
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- Eval methodCheck LLM state policy recall for cross-jurisdiction substitutionVerify if an LLM provides another state's policy value when asked about a specific state. · 60 min
- Troubleshooting fixCorrect LLM state policy errors by checking other states' valuesWhen an LLM gives a wrong state policy answer, check if it matches another state's correct value. · 10 min
- ChecklistCheck AI model safety evidence acquisition policiesEvaluate how AI models seek safety information before acting using a controlled benchmark. · 120 min
- ChecklistCoE Audit integrity checks for AI-generated papersAutomated checks for references, scores, methods, and code alignment in AI research outputs. · 10 min
- Eval methodMethod for evaluating potential discriminatory impact of LMsUse an LM to generate prompts across scenarios, varying demographics to find discrimination. · 60 min
- Troubleshooting fixDecrease LM discrimination via careful prompt engineeringUse prompt engineering to significantly decrease positive and negative discrimination in LM outputs. · 20 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- arXiv (arXiv cs.AI). Subvolts collected, normalized and credited it; the link above opens the original.