mlsanityData to 7 Oct 2026

Unified Pre-training for Program Understanding and Generation

arXiv 2103.06333 ↗First posted 10 Mar 2021cs.CL cs.PLcited by 361

Not on the map. Why?

Abstract (arXiv)

Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.

Key facts. Cited by 361 papers in the map (data through 7 Oct 2026). Builds on 40 of its 63 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

40 of 63 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
28 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  3. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  4. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  5. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  7. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  8. 2020Unsupervised Translation of Programming Languagescited by 262
  9. 2018code2vec: Learning Distributed Representations of Codecited by 249
  10. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  11. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  12. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  13. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  14. 2018Automated Vulnerability Detection in Source Code Using Deep Representation Learningcited by 114
  15. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  16. 2018Mapping Language to Code in Programmatic Contextcited by 104
  17. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  18. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  19. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  20. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 68
  21. 2017Code Completion with Neural Attention and Pointer Networkscited by 49
  22. 2018CODIT: Code Editing with Tree-Based Neural Modelscited by 40
  23. 2015On the "Naturalness" of Buggy Codecited by 35
  24. 2019Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Datacited by 20
  25. 2019Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsingcited by 18
  26. 2019Learning Programmatic Idioms for Scalable Semantic Parsingcited by 17
  27. 2018Building Language Models for Text with Named Entitiescited by 15
  28. 2018Program Language Translation Using a Grammar-Driven Tree-to-Tree Modelcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 77; 2023: 118; 2024: 71; 2025: 42; 2026: 24.

361 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  3. 2023StarCoder: may the source be with you!cited by 790
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  7. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  8. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  10. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  11. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  12. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
38 more
  1. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  2. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  3. 2023Planning with Large Language Models for Code Generationcited by 179
  4. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  5. 2023SantaCoder: don't reach for the stars!cited by 169
  6. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  7. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  8. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  9. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  10. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  11. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  12. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  13. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  14. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  15. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
  16. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  17. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  18. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  19. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  20. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  21. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  22. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  23. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  24. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  25. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  26. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  27. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  28. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  29. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  30. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  31. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  32. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  33. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  34. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  35. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  36. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  37. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 54
  38. 2022When Language Model Meets Private Librarycited by 54

50 most cited of 361 All 361 (JSON)