# ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

arXiv: https://arxiv.org/abs/2609.18805 · HTML: https://mlsanity.com/paper/2609.18805/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 23 of 23)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs](https://mlsanity.com/paper/2406.20098/) (2024)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping](https://mlsanity.com/paper/2411.03292/) (2024)
- [DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks](https://mlsanity.com/paper/2607.07946/) (2026)
- [BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills](https://mlsanity.com/paper/2510.19898/) (2025)
- [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://mlsanity.com/paper/2606.29957/) (2026)
- [Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification](https://mlsanity.com/paper/2603.26648/) (2026)
- [Test Oracle Automation in the era of LLMs](https://mlsanity.com/paper/2405.12766/) (2024)
- [MirrorCode: AI can rebuild entire programs from behavior alone](https://mlsanity.com/paper/2606.30182/) (2026)
- [SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?](https://mlsanity.com/paper/2608.19799/) (2026)
- [VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents](https://mlsanity.com/paper/2605.26144/) (2026)

## Abstract (arXiv)

Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.