Which AI tools does this work with?

Any current model. It was seen with in the sources.

Where did this come from?

Google Research (Google Research Blog). Subvolts collected, normalized and credited it; the link above opens the original.

Checklist Analysis and data ResearchExpert · 30 min

LLM knowledge profiling: encoding and recall assessment

Checklist for assessing LLM factuality by measuring encoding and recall.

Any modelParaphrased

Items

Open in ChatGPT Open in Claude

Measure fact encoding by prompting the LLM to reproduce facts within their original context.
Measure knowledge recall by asking questions across varied phrasings and relational directions.
Evaluate recall with and without 'thinking' prompts (e.g., chain-of-thought).
Classify each fact into one of five knowledge profiles: encoding failure, recall failure, direct recall, recall with thinking, or inference without encoding.
Analyze the distribution of knowledge profiles to identify bottlenecks (encoding vs. recall).

Measure fact encoding by prompting the LLM to reproduce facts within their original context.
Measure knowledge recall by asking questions across varied phrasings and relational directions.
Evaluate recall with and without 'thinking' prompts (e.g., chain-of-thought).
Classify each fact into one of five knowledge profiles: encoding failure, recall failure, direct recall, recall with thinking, or inference without encoding.
Analyze the distribution of knowledge profiles to identify bottlenecks (encoding vs. recall).

Before or after

after

Context

Evaluating the factuality and knowledge retrieval capabilities of Large Language Models.

What it does and when to use it

A checklist for assessing LLM factuality by measuring how well models encode and recall information. It involves specific prompting techniques for encoding and recall, followed by classifying facts into knowledge profiles.

Why it works

This method provides a more granular understanding of LLM limitations by separating encoding from recall, offering insights into whether errors are due to missing knowledge or retrieval issues.

Sources and credit

Empty shelves or lost keys? Recall is the bottleneck for parametric factuality ↗Google Research · Google Research Blog · Aug 12, 2026 · paraphrased

First seen on Subvolts Aug 12, 2026. Updated Sep 11, 2026. Spotted a problem? Tell us.

Where it shows up

Empty shelves or lost keys? Recall is the bottleneck for parametric factualityGoogle · Google Research Blog · Aug 12, 2026

Terms used

Encoding Recall

Related in the PlaybookPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

ToolSciTrue for scientific claim validation using LLMsSystem for validating scientific claims against tables and figures in papers using frontier and open language models. · 10 min
Anti-patternRelying on static benchmarks in web-enabled LLM environmentsStatic benchmarks are unreliable when LLMs can access and analyze web content during evaluation. · 10 min
Eval methodEvaluate LLM forecasting performance with structured effectsMeasure how much structured future information improves LLM time series forecasting accuracy. · 30 min
ChecklistCheck AI model safety evidence acquisition policiesEvaluate how AI models seek safety information before acting using a controlled benchmark. · 120 min
ChecklistAudit LLM evaluation for citation artifacts and biasCheck for presentation cues, citation issues, and quality gaps in LLM-as-a-judge evaluations. · 30 min
Troubleshooting fixUnderestimated standard error in non-independent eval questionsUse clustered standard errors when eval questions are related to avoid underestimating uncertainty and drawing incorrect conclusions. · 5 min

Questions people ask

Which AI tools does this work with?: Any current model. It was seen with in the sources.
Where did this come from?: Google Research (Google Research Blog). Subvolts collected, normalized and credited it; the link above opens the original.

LLM knowledge profiling: encoding and recall assessment

Questions people ask

OpenAI (ChatGPT)

Google (Gemini)

Anthropic (Claude)

Microsoft (Copilot)

Meta (Meta AI)

xAI (Grok)

Across the six

Subvolts