mlsanityData to 7 Oct 2026

A Survey on Pretrained Language Models for Neural Code Intelligence

arXiv 2212.10079 ↗First posted 20 Dec 2022cs.SE cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

As the complexity of modern software continues to escalate, software engineering has become an increasingly daunting and error-prone endeavor. In recent years, the field of Neural Code Intelligence (NCI) has emerged as a promising solution, leveraging the power of deep learning techniques to tackle analytical tasks on source code with the goal of improving programming efficiency and minimizing human errors within the software industry. Pretrained language models have become a dominant force in NCI research, consistently delivering state-of-the-art results across a wide range of tasks, including code summarization, generation, and translation. In this paper, we present a comprehensive survey of the NCI domain, including a thorough review of pretraining techniques, tasks, datasets, and model architectures. We hope this paper will serve as a bridge between the natural language and programming language communities, offering insights for future research in this rapidly evolving field.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 48 of its 78 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

48 of 78 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
36 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  4. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  7. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  9. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  11. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  12. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  13. 2020Unsupervised Translation of Programming Languagescited by 262
  14. 2018code2vec: Learning Distributed Representations of Codecited by 249
  15. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  16. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  17. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  18. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  19. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  20. 2018Mapping Language to Code in Programmatic Contextcited by 104
  21. 2020Contrastive Code Representation Learningcited by 76
  22. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  23. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  24. 2018Neural Code Comprehension: A Learnable Representation of Code Semanticscited by 59
  25. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  26. 2021D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysiscited by 48
  27. 2019Recommendations for Datasets for Source Code Summarizationcited by 46
  28. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  29. 2021How could Neural Networks understand Programs?cited by 26
  30. 2020Deep Graph Matching and Searching for Semantic Code Retrievalcited by 20
  31. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18
  32. 2021Solving Linear Algebra by Program Synthesiscited by 14
  33. 2022Automatically Generating CS Learning Materials with Large Language Modelscited by 12
  34. 2021Solving Probability and Statistics Problems by Program Synthesiscited by 9
  35. 2018Program Synthesis Through Reinforcement Learning Guided Tree Searchcited by 7
  36. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 4; 2025: 2.

13 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  3. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  4. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  5. 2023Scope is all you need: Transforming LLMs for HPC Codecited by 12
  6. 2023MonoCoder: Domain-Specific Code Language Model for HPC Codes and Taskscited by 12
  7. 2025Towards an Understanding of Context Utilization in Code Intelligencecited by 10
  8. 2023TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skillscited by 8
  9. 2023Lingua Manga: A Generic Large Language Model Centric System for Data Curationcited by 8
  10. 2024GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Modelscited by 6
  11. 2025MiniF2F in Rocq: Automatic Translation Between Proof Assistants -- A Case Studycited by 5
  12. 2024Brevity is the soul of wit: Pruning long files for code generationcited by 4
1 more
  1. 2024Structure-aware Fine-tuning for Code Pre-trained Modelscited by 3