AX Practitioner Playbook Evaluation Quality Checklist
Review your AI agent evaluation process before running it to ensure its quality and trustworthiness.
before
Ensuring the quality of an AI agent evaluation before execution.
Before running an AI agent evaluation, use this checklist to ensure your setup, criteria, and testing plans are robust. It helps confirm that your evaluation process is reliable and will yield trustworthy results.
This checklist ensures all critical aspects of an AI agent evaluation are considered, preventing common pitfalls and increasing the likelihood of identifying accurate issues and implementing effective fixes.
When conducting a casual, low-stakes test of an AI agent's capabilities.
- Introducing the Agent Experience (AX) Practitioner Playbook ↗Microsoft · Microsoft for Developers · Oct 9, 2026 · paraphrased
- How to make AI coding agents better at using your technology ↗Microsoft · Microsoft Source · Oct 9, 2026 · paraphrased
First seen on Subvolts Oct 9, 2026. Updated Oct 9, 2026. 2 sources describe this; the earliest is credited first. Spotted a problem? Tell us.
- How to make AI coding agents better at using your technologyMicrosoft · Microsoft Source · Oct 9, 2026
- Introducing the Agent Experience (AX) Practitioner PlaybookMicrosoft · Microsoft for Developers · Oct 9, 2026
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- Anti-patternLetting Models Write Evaluation CriteriaAvoid having AI models generate your evaluation criteria for AI agent performance. · 10 min
- WorkflowBuild and test an AI agent using Microsoft Foundry ToolkitCreate, configure, test, and evaluate an AI agent within VS Code. · 60 min
- ChecklistChecklist for production-ready AI agent refactoringSteps to ensure an AI agent is production-ready after refactoring. · 20 min
- HeuristicDon't Wait for Models to Improve for Your TechActively improve AI agent performance by controlling their information sources. · 15 min
- ToolMicrosoft Foundry Toolkit for building AI agents in VS CodeBuild, test, and deploy AI agents directly within your VS Code environment. · 30 min
- WorkflowUse agent apps to scope, secure, roll out, and ship a featureLeverage GitHub agent apps to analyze data, check dependencies, set feature flags, and assess deploy risk. · 30 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with Copilot in the sources.
- Where did this come from?
- Microsoft (Microsoft for Developers); Microsoft (Microsoft Source). Subvolts collected, normalized and credited it; the link above opens the original.