Pass@k metric
Also written: pass@k
Metric for code generation: checks if a correct solution is in the top k outputs.
Pass@k measures how well a model can generate correct code. Instead of just looking at the single best code snippet a model produces, Pass@k checks if the right code is found within the top 'k' guesses.
For example, if a problem asks for a Python function to add two numbers, and the model generates 10 different functions (k=10), Pass@10 would be considered successful if at least one of those 10 functions correctly adds two numbers. This is useful because sometimes a model might not produce the perfect answer first, but a correct one is still among its close attempts.
Pass@k is a benchmark metric for evaluating code generation models. It measures the probability that at least one of the top 'k' code samples generated by the model for a given problem is a correct solution. This is typically calculated by generating 'm' samples for each of 'n' problems, and then checking if any of the 'm' samples for a problem pass the unit tests. The metric is often reported as Pass@k, where k is the number of samples considered (e.g., Pass@1, Pass@10, Pass@100).
Pass@k is relevant as frontier AI labs focus on improving code generation capabilities. Google Gemini's research on reasoning suggests that better reasoning can lead to more accurate outputs, which directly impacts metrics like Pass@k by increasing the likelihood of correct solutions appearing in generated samples.
- Thinking to recall: How reasoning unlocks parametric knowledge in LLMsGoogle · Google Research Blog ·
- Google Gemini
- Google Gemini research explores how reasoning improves LLM performance, which is critical for metrics like Pass@k used to evaluate code generation.
- Accuracy
- Accuracy typically measures the proportion of correct predictions out
- BLEU score
- BLEU is a metric for evaluating machine translation quality by compa
The Pass@k metric was introduced in a 2021 paper by researchers from Microsoft and the University of California, Berkeley, to evaluate code generation models. Source ↗
Ask a model like ChatGPT or Google Gemini to write a simple Python function, such as 'write a Python function to calculate the factorial of a number'. Then, try running the generated code with a few test cases to see if it works correctly. You can try asking for multiple versions to see if one works.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- What does 'k' mean in Pass@k?
- The 'k' in Pass@k represents the number of code samples the model generates for a single problem. Pass@k checks if any of these top 'k' samples are correct.
- Why is Pass@k better than just checking the single best answer?
- Pass@k is more forgiving and reflects real-world use where a user might try a few options. It acknowledges that a model might produce a correct solution within its top attempts, even if not the absolute first one.
First seen on Subvolts Jun 24, 2026. Updated Sep 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.