mlsanityData to 14 Oct 2026

CoditT5: Pretraining for Source Code and Natural Language Editing

arXiv 2208.05446 ↗First posted 10 Aug 2022cs.SE cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Pretrained language models have been shown to be effective in many software-related generation tasks; however, they are not well-suited for editing tasks as they are not designed to reason about edits. To address this, we propose a novel pretraining objective which explicitly models edits and use it to build CoditT5, a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments. We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review. By outperforming standard generation-based models, we demonstrate the generalizability of our approach and its suitability for editing tasks. We also show how a standard generation model and our edit-based model can complement one another through simple reranking strategies, with which we achieve state-of-the-art performance for the three downstream editing tasks.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 44 of its 64 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

44 of 64 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  11. 2015Pointer Networkscited by 895
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
32 more
  1. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  2. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  3. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  4. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  5. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  6. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 218
  7. 2019Strategies for Structuring Story Generationcited by 124
  8. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 108
  9. 2017Generating Sentences by Editing Prototypescited by 88
  10. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  11. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  12. 2021Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Taskscited by 70
  13. 2017Event Representations for Automated Story Generation with Deep Neural Netscited by 63
  14. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  15. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  16. 2021Towards Automating Code Review Activitiescited by 54
  17. 2019On Learning Meaningful Code Changes via Neural Machine Translationcited by 50
  18. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  19. 2018Learning to Represent Editscited by 34
  20. 2020Learning to Update Natural Language Comments Based on Code Changescited by 31
  21. 2019Learning to Fix Build Errors with Graph2Diff Neural Networkscited by 26
  22. 2021On Multi-Modal Learning of Editing Source Codecited by 25
  23. 2019Complexity-Weighted Loss and Diverse Reranking for Sentence Simplificationcited by 20
  24. 2020Deep Just-In-Time Inconsistency Detection Between Comments and Source Codecited by 20
  25. 2020Patching as Translation: the Data and the Metaphorcited by 18
  26. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18
  27. 2021Learning Structural Edits via Incremental Tree Transformationscited by 16
  28. 2020Copy that! Editing Sequences by Copying Spanscited by 13
  29. 2021Ensemble Models for Neural Source Code Summarization of Subroutinescited by 12
  30. 2018Sequential Copying Networkscited by 10
  31. 2021Automating the Removal of Obsolete TODO Commentscited by 6
  32. 2015Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 23; 2024: 9; 2025: 6; 2026: 6.

45 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  3. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  4. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  5. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  6. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 69
  7. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  8. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 48
  9. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  10. 2024Automatic Programming: Large Language Models and Beyondcited by 36
  11. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  12. 2023Learning Deep Semantics for Test Completioncited by 31
33 more
  1. 2023Better patching using LLM prompting, via Self-Consistencycited by 29
  2. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 28
  3. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  4. 2023Text-to-SQL Error Correction with Language Models of Codecited by 21
  5. 2023GrACE: Generation using Associated Code Editscited by 16
  6. 2023Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptionscited by 14
  7. 2024Diffusion On Syntax Trees For Program Synthesiscited by 10
  8. 2023Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editingcited by 9
  9. 2023Multilingual Code Co-Evolution Using Large Language Modelscited by 9
  10. 2023Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Codecited by 9
  11. 2024Automated Program Repair: Emerging trends pose and expose problems for benchmarkscited by 7
  12. 2024Memory-Efficient Large Language Models for Program Repair with Semantic-Guided Patch Generationcited by 5
  13. 2024exLong: Generating Exceptional Behavior Tests with Large Language Modelscited by 4
  14. 2024Improving Retrieval-Augmented Code Comment Generation by Retrieving for Generationcited by 4
  15. 2025Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction Historycited by 3
  16. 2023Interactive Text Generationcited by 2
  17. 2023Coarse-Tuning Models of Code with Reinforcement Learning Feedbackcited by 2
  18. 2025LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperationcited by 2
  19. 2025NES: An Instruction-Free, Low-Latency Next Edit Suggestion Framework Powered by Learned Historical Editing Trajectoriescited by 2
  20. 2026To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editingcited by 2
  21. 2023Towards Automatically Addressing Self-Admitted Technical Debt: How Far Are We?cited by 1
  22. 2024Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learningcited by 1
  23. 2025DSBC : Data Science task Benchmarking with Context engineeringcited by 1
  24. 2026Pull Requests as a Training Signal for Repo-Level Code Editingcited by 1
  25. 2026Learning from 53.6K Real-World Developer Edits of AI-Generated Codecited by 1
  26. 2023On ML-Based Program Translation: Perils and Promisescited by 0
  27. 2023Toward Interactive Dictationcited by 0
  28. 2024Linguacodus: A Synergistic Framework for Transformative Code Generation in Machine Learning Pipelinescited by 0
  29. 2025Wired for Reuse: Automating Context-Aware Code Adaptation in IDEs via LLM-Based Agentcited by 0
  30. 2025What a diff makes: automating code migration with large language modelscited by 0
  31. 2026Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Rewardcited by 0
  32. 2026MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugscited by 0
  33. 2026CROCODIL: Cross-Model Code Editing with LLMscited by 0