mlsanityData to 7 Oct 2026

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

arXiv 2109.00859 ↗First posted 2 Sep 2021cs.CL cs.PLcited by 890

Not on the map. Why?

Cited by 61 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https: //github.com/salesforce/CodeT5 .

Key facts. Cited by 890 papers in the map (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 2 Sep 2021.

Builds on

30 of 38 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  12. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
18 more
  1. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  2. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  3. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  4. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  5. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  6. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  7. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  8. 2020Unsupervised Translation of Programming Languagescited by 262
  9. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  10. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  11. 2018Mapping Language to Code in Programmatic Contextcited by 104
  12. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  13. 2021Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Taskscited by 70
  14. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 68
  15. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  16. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  17. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  18. 2021CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 104; 2023: 237; 2024: 226; 2025: 213; 2026: 106.

890 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2024A Survey on Large Language Models for Code Generationcited by 646
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  3. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  4. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  5. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  6. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  9. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  10. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  12. 2023A Survey on Model Compression for Large Language Modelscited by 263
  13. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  14. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  15. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  16. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  17. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  18. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  19. 2023Planning with Large Language Models for Code Generationcited by 179
  20. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  21. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  22. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  23. 2023SantaCoder: don't reach for the stars!cited by 169
  24. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  25. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  26. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  27. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  28. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  29. 2023Auditing large language models: a three-layered approachcited by 135
  30. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  31. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 132
  32. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  33. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  34. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  35. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  36. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  37. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  38. 2024Large Language Models Meet NLP: A Surveycited by 102

50 most cited of 890 All 890 (JSON)