mlsanityData to 7 Oct 2026

Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code

arXiv 2205.11739 ↗First posted 24 May 2022cs.SE cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Recent years have seen the successful application of deep learning to software engineering (SE). In particular, the development and use of pre-trained models of source code has enabled state-of-the-art results to be achieved on a wide variety of SE tasks. This paper provides an overview of this rapidly advancing field of research and reflects on future research directions.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

56 of 79 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
38 more
  1. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  2. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  3. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  4. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  5. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  6. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  7. 2020Unsupervised Translation of Programming Languagescited by 262
  8. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  9. 2015Semi-supervised Sequence Learningcited by 198
  10. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  11. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  12. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  13. 2020Improved Code Summarization via a Graph Neural Networkcited by 79
  14. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  15. 2020Contrastive Code Representation Learningcited by 76
  16. 2021Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Taskscited by 70
  17. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 68
  18. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  19. 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
  20. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  21. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  22. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  23. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  24. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  25. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 56
  26. 2019On Learning Meaningful Code Changes via Neural Machine Translationcited by 50
  27. 2020On Learning Meaningful Assert Statements for Unit Test Casescited by 42
  28. 2021TreeBERT: A Tree-Based Pre-Trained Model for Programming Languagecited by 40
  29. 2019Neural Program Repair by Jointly Learning to Localize and Repaircited by 39
  30. 2022SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representationscited by 35
  31. 2020Improved Automatic Summarization of Subroutines via Attention to File Contextcited by 34
  32. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  33. 2021ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generationcited by 31
  34. 2020Syntax-Enhanced Pre-trained Modelcited by 29
  35. 2019Automatic Source Code Summarization with Extended Tree-LSTMcited by 28
  36. 2021How could Neural Networks understand Programs?cited by 26
  37. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18
  38. 2019Learning Programmatic Idioms for Scalable Semantic Parsingcited by 17

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 4; 2025: 5; 2026: 1.

20 papers.

Sort:

  1. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  2. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  3. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  4. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  5. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  6. 2024A Large-Scale Exploit Instrumentation Study of AI/ML Supply Chain Attacks in Hugging Face Modelscited by 12
  7. 2022CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structurecited by 6
  8. 2023The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classificationcited by 5
  9. 2022A Survey on Natural Language Processing for Programmingcited by 3
  10. 2023Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?cited by 3
  11. 2024Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translationcited by 3
  12. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
8 more
  1. 2025Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforcescited by 2
  2. 2023An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Codecited by 1
  3. 2025Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findingscited by 1
  4. 2023DocGen: Generating Detailed Parameter Docstrings in Pythoncited by 0
  5. 2024Multi-modal Learning for WebAssembly Reverse Engineeringcited by 0
  6. 2025LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Modelscited by 0
  7. 2025HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detectioncited by 0
  8. 2026CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generationcited by 0