Which AI tools does this work with?

Any current model. It was seen with in the sources.

Where did this come from?

OpenAI (OpenAI News). Subvolts collected, normalized and credited it; the link above opens the original.

Checklist Analysis and data Research Planning and PMExpert · 15 min

Evaluate AI model claims with detailed setup descriptions

Ensure AI model evaluations clearly state claims tested and provide evidence for result validity.

Any modelParaphrased

Items

Open in ChatGPT Open in Claude

Specify the claim the evaluation setup was designed to test.
Share available evidence that the evaluation result is valid.
Describe the harness used, including tools, scaffolding, and budget.
Explain how evaluators checked for effects that could impact result validity.
Detail the elicitation strategy and adversary model for safeguard robustness tests.
Report performance under specific harness and budget conditions, not as a capability ceiling.
Consider expected cost per successful solve where applicable.

Specify the claim the evaluation setup was designed to test.
Share available evidence that the evaluation result is valid.
Describe the harness used, including tools, scaffolding, and budget.
Explain how evaluators checked for effects that could impact result validity.
Detail the elicitation strategy and adversary model for safeguard robustness tests.
Report performance under specific harness and budget conditions, not as a capability ceiling.
Consider expected cost per successful solve where applicable.

Before or after

before

Context

Evaluating AI model capabilities and safeguards.

What it does and when to use it

This checklist ensures AI model evaluations are robust and interpretable. It guides evaluators to clearly define the claims being tested, provide evidence for the validity of results, and detail the 'harness' or setup used. This promotes transparency and allows readers to understand how results depend on the testing environment.

Why it works

Clear descriptions of claims and evidence, along with detailed setup information, make evaluation results more trustworthy and understandable. Understanding the 'harness' is crucial because it significantly impacts model performance, especially for complex, multi-step tasks.

Sources and credit

A shared playbook for trustworthy third party evaluations ↗OpenAI · OpenAI News · May 28, 2026 · paraphrased

First seen on Subvolts May 28, 2026. Updated Sep 11, 2026. Spotted a problem? Tell us.

Where it shows up

A shared playbook for trustworthy third party evaluationsOpenAI · OpenAI News · May 28, 2026

Related in the PlaybookPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Eval methodMethod for evaluating potential discriminatory impact of LMsUse an LM to generate prompts across scenarios, varying demographics to find discrimination. · 60 min
ChecklistAI Model Transparency and Safety Evaluation ChecklistChecklist for assessing AI model transparency and safety trade-offs. · 45 min
ChecklistSafety and feasibility assessment for clinical AI deploymentKey considerations for evaluating AI in clinical settings. · 15 min
Prompt patternTemplate for clinical decision support promptsA flexible template for creating clinical decision support prompts for various patient scenarios. · 5 min
Anti-patternMisconfiguring AI environments for cybersecurity evaluationsDo not tell models they are in a simulation if they are connected to the real internet. · 5 min
Case studyUsing Screwtape to find confirmed failures of AI monitorsDemonstrates how a red-teamer can find confirmed failures of an AI monitor without developer infrastructure access. · 120 min

Questions people ask

Which AI tools does this work with?: Any current model. It was seen with in the sources.
Where did this come from?: OpenAI (OpenAI News). Subvolts collected, normalized and credited it; the link above opens the original.

Evaluate AI model claims with detailed setup descriptions

Questions people ask

OpenAI (ChatGPT)

Google (Gemini)

Anthropic (Claude)

Microsoft (Copilot)

Meta (Meta AI)

xAI (Grok)

Across the six

Subvolts