Learning from 53.6K Real-World Developer Edits of AI-Generated Code
Not on the map. Why?
Abstract (arXiv)
Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant. Manual code edits provide more realistic and granular information on editing behavior than Git commits, which only contain final successful code snippets. Yet, due to a lack of high-quality, realistic code editing data, LLMs are mostly trained on publicly available Git data (e.g., commits). To address this gap, we introduce DECODE (Developer Edits of Code Dataset), a dataset of 53.6K real-world in-IDE code edits of AI-generated code in Python, TypeScript, and JavaScript, sourced from 1K+ developers. First, we demonstrate the utility of DECODE for data analysis, obtaining insights on when, why, and how AI-generated code is edited. We find that most edits occur within the first 15 minutes after accepting an AI completion, resulting in the removal of AI completions in 31% of edit trajectories. Second, we use DECODE to benchmark the ability of LLMs to predict code edits. We find that finetuning on DECODE enables open-source 3B models to perform code edit prediction tasks significantly better than frontier LLMs. We then discuss implications of this work, emphasizing the necessity of developer-centric machine learning approaches for future AI programming assistants.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 27 Jul 2026.
Builds on
43 of 59 references.
References · 43
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
- 2025OpenAI GPT-5 System Cardcited by 989
- 2023StarCoder: may the source be with you!cited by 799
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
31 more
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
- 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
- 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
- 2023Learning Performance-Improving Code Editscited by 106
- 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
- 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
- 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
- 2026Qwen3-Coder-Next Technical Reportcited by 71
- 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 48
- 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
- 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 43
- 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 29
- 2025Copilot Arena: A Platform for Code LLM Evaluation in the Wildcited by 26
- 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 25
- 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
- 2024The Use of Generative Search Engines for Knowledge Work and Complex Taskscited by 18
- 2023GrACE: Generation using Associated Code Editscited by 16
- 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
- 2025EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Editscited by 13
- 2022CodeEditor: Learning to Edit Source Code with Pre-trained Modelscited by 12
- 2025Modeling Student Learning with 3.8 Million Program Tracescited by 10
- 2024Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Codecited by 9
- 2022Overwatch: Learning Patterns in Code Edit Sequencescited by 8
- 2023A Qualitative Study on the Implementation Design Decisions of Developerscited by 7
- 2026Learning Next Action Predictors from Human-Computer Interactioncited by 7
- 2019How Different Are Different diff Algorithms in Git?cited by 6
- 2024Retrospective Learning from Interactionscited by 6
- 2025Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction Historycited by 3
- 2025Generating High-Quality Datasets for Code Editing via Open-Source Language Modelscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.