mlsanityData to 7 Oct 2026

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

arXiv 2305.07922 ↗First posted 13 May 2023cs.CL cs.LG cs.PLcited by 327

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations in terms of architecture and pretraining tasks. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks. The former paradigm is limited by inflexibility in applications while in the latter, the model is treated as a single system for all tasks, leading to suboptimal performance on a subset of tasks. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some downstream tasks and hence result in substantial performance degrade. To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives to mitigate the pretrain-finetune discrepancy. These objectives cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) model performance on various code-related tasks, such as code generation and completion, math programming, and text-to-code retrieval tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA results on HumanEval code generation task against other open code LLMs.

Key facts. Cited by 327 papers in the map (data through 7 Oct 2026). Builds on 52 of its 71 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

52 of 71 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
  13. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  14. 2023StarCoder: may the source be with you!cited by 790
  15. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  16. 2021GPT Understands, Toocited by 715
  17. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  18. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  19. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  20. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  21. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  22. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  23. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  24. 2022CodeT: Code Generation with Generated Testscited by 369
  25. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  26. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  27. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  28. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  29. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  30. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  31. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  32. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  33. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  34. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  35. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  36. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  37. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  38. 2022SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representationscited by 35

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 65; 2024: 126; 2025: 95; 2026: 41.

327 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  6. 2023Textbooks Are All You Needcited by 561
  7. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  8. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  9. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  10. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  11. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  12. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
38 more
  1. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  2. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  3. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  4. 2025MemOS: A Memory OS for AI Systemcited by 129
  5. 2024Large Language Models Meet NLP: A Surveycited by 102
  6. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  7. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  8. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  9. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  10. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  11. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  12. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  13. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  14. 2024Instruction Tuning for Secure Code Generationcited by 63
  15. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  16. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  17. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  18. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  19. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  20. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  21. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  22. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  23. 2023Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectivescited by 48
  24. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  25. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 45
  26. 2023RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repaircited by 43
  27. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  28. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  29. 2024LeDex: Training LLMs to Better Self-Debug and Explain Codecited by 39
  30. 2024How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmarkcited by 39
  31. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  32. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  33. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  34. 2023Rethinking the Instruction Quality: LIFT is What You Needcited by 33
  35. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 32
  36. 2024An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detectioncited by 32
  37. 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Modelscited by 32
  38. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31

50 most cited of 327 All 327 (JSON)