Inferring behavior from working software for coding agents
Coding agents should be evaluated on their ability to infer behavior from running applications, not just explicit instructions.
Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application.
This approach moves beyond simple instruction following to assess an agent's capacity for understanding and replicating functionality observed in a live system, which is more representative of real-world development challenges.
An agent is given a partially built e-commerce site and a fully functional reference site. The agent must interact with the reference site to understand how features like 'add to cart' or 'checkout' work, then implement them in the incomplete application.
Evaluate coding agents based on their ability to infer functionality from working software examples, rather than solely relying on explicit instructions or issue tickets.
- ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks ↗arXiv · arXiv cs.AI · Sep 16, 2026 · quoted as shared
First seen on Subvolts Sep 16, 2026. Updated Sep 18, 2026. Spotted a problem? Tell us.
- ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE TasksAnthropic · arXiv cs.AI · Sep 16, 2026
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- HeuristicUse issue trackers as control planes for AI agentsLeverage your issue tracker as a central control plane to manage and orchestrate AI agentic work. · 10 min
- HeuristicConsider engineering constraints beyond functional correctness for AI agentsAI coding agents must follow developer-defined constraints like scope, order, validation, tool use, and reporting, not just produce working code. · 5 min
- ChecklistSWE-Duel Agent Evaluation CriteriaCriteria for admitting a pull request and for a Blue agent to win in the SWE-Duel arena. · 5 min
- Cost tipReduce AI coding costs with CliffCompactionCut AI coding costs up to 50% for long tasks by compacting context without rephrasing. · 15 min
- ToolGameLogicBench for evaluating AI game logic codingA benchmark to evaluate AI agents on runtime game logic with tick-level state assertions in Godot projects. · 10 min
- HeuristicSelf-Scaling Benchmarks for Coding AgentsBenchmarks should create tasks at evaluation time to prevent data leakage and adapt difficulty based on agent performance. · 5 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- arXiv (arXiv cs.AI). Subvolts collected, normalized and credited it; the link above opens the original.