# EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits

arXiv: https://arxiv.org/abs/2511.04486 · HTML: https://mlsanity.com/paper/2511.04486/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2025.

## Built on by (most cited first, 13 of 13)

- [GameDevBench: Evaluating Agentic Capabilities Through Game Development](https://mlsanity.com/paper/2602.11103/) (2026)
- [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://mlsanity.com/paper/2606.29957/) (2026)
- [Corrective Diffusion Language Models](https://mlsanity.com/paper/2512.15596/) (2025)
- [KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?](https://mlsanity.com/paper/2601.13240/) (2026)
- [SWE-Touch: Benchmarking Coding Agents When Users Touch the Code](https://mlsanity.com/paper/2608.02499/) (2026)
- [Humans are Missing from AI Coding Agent Research](https://mlsanity.com/paper/2608.12355/) (2026)
- [When Models Edit Too Much: On the Fidelity of Minimal Code Edits](https://mlsanity.com/paper/2609.04061/) (2026)
- [Comparing Developer and LLM Biases in Code Evaluation](https://mlsanity.com/paper/2603.24586/) (2026)
- [Learning from 53.6K Real-World Developer Edits of AI-Generated Code](https://mlsanity.com/paper/2607.25130/) (2026)
- [To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing](https://mlsanity.com/paper/2607.28887/) (2026)
- [SemRep: Generative Code Representation Learning with Code Transformations](https://mlsanity.com/paper/2603.13640/) (2026)
- [RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications](https://mlsanity.com/paper/2607.06411/) (2026)
- [CROCODIL: Cross-Model Code Editing with LLMs](https://mlsanity.com/paper/2609.03894/) (2026)

## Abstract (arXiv)

Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. However, few benchmarks directly evaluate this capability and current datasets often rely on artificial sources. We introduce EDIT-Bench, a benchmark for evaluating LLM code editing capabilities grounded in real-world usage, i.e., user instructions and code contexts collected in the wild. EDIT-Bench comprises of 540 problems, multiple natural and programming languages, and a diverse set of real-world use cases, ranging from resolving errors to adding features. EDIT-Bench introduces context-dependent problems that require the model to understand code context, highlighted code, and cursor position in addition to the user instruction. We evaluate 40 diverse LLMs and observe that EDIT-Bench is a challenging set of problems where only 1 model scores over 60%. We find that model performance varies across different categories of user instructions. Further, we find that varying levels of contextual information greatly affect task success rate, with performance varying up to 11%, indicating the importance of evaluating with realistic context.