mlsanityData to 7 Oct 2026

Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review

arXiv 2307.02503 ↗First posted 4 Jul 2023cs.SE cs.AI cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

This paper provides a comprehensive review of the literature concerning the utilization of Natural Language Processing (NLP) techniques, with a particular focus on transformer-based large language models (LLMs) trained using Big Code, within the domain of AI-assisted programming tasks. LLMs, augmented with software naturalness, have played a crucial role in facilitating AI-assisted programming applications, including code generation, code completion, code translation, code refinement, code summarization, defect detection, and clone detection. Notable examples of such applications include the GitHub Copilot powered by OpenAI's Codex and DeepMind AlphaCode. This paper presents an overview of the major LLMs and their applications in downstream tasks related to AI-assisted programming. Furthermore, it explores the challenges and opportunities associated with incorporating NLP techniques with software naturalness in these applications, with a discussion on extending AI-assisted programming capabilities to Apple's Xcode for mobile software development. This paper also presents the challenges of and opportunities for incorporating NLP techniques with software naturalness, empowering developers with advanced coding assistance and streamlining the software development process.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 103 of its 212 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

103 of 212 references.

References · 103

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2022Generative Adversarial Networkscited by 5,603
  12. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
38 more
  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  12. 2020Reformer: The Efficient Transformercited by 1,658
  13. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  14. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  15. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  16. 2021Measuring Coding Challenge Competence With APPScited by 879
  17. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  18. 2023StarCoder: may the source be with you!cited by 790
  19. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  20. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  21. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  22. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  23. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  24. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  25. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  26. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  27. 2022CodeT: Code Generation with Generated Testscited by 369
  28. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  29. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  30. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  31. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  32. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  33. 2020Unsupervised Translation of Programming Languagescited by 262
  34. 2018code2vec: Learning Distributed Representations of Codecited by 249
  35. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  36. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  37. 2017Learning to Represent Programs with Graphscited by 219
  38. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 10; 2025: 12; 2026: 4.

32 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  4. 2025CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generationcited by 65
  5. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  6. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  7. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  8. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  9. 2024Materials science in the era of large language models: a perspectivecited by 23
  10. 2023Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Modelscited by 11
  11. 2025ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Modelscited by 10
  12. 2025StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysiscited by 8
20 more
  1. 2025AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generationcited by 8
  2. 2025The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impairedcited by 7
  3. 2023Exploring Large Language Models for Code Explanationcited by 6
  4. 2026Towards High-Fidelity CAD Generation via LLM-Driven Program Generation and Text-Based B-Rep Primitive Groundingcited by 6
  5. 2023Large Language Model-Driven Classroom Flipping: Empowering Student-Centric Peer Questioning with Flipped Interactioncited by 5
  6. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  7. 2023Can LLMs Patch Security Issues?cited by 3
  8. 2025Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Modelscited by 3
  9. 2025Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questionscited by 3
  10. 2026Why Human Guidance Matters in Collaborative Vibe Codingcited by 3
  11. 2024Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programmingcited by 2
  12. 2024Optimizing AI-Assisted Code Generationcited by 2
  13. 2025Reverse Engineering User Stories from Code using Large Language Modelscited by 2
  14. 2024A GPT-based Code Review System for Programming Language Learningcited by 1
  15. 2025From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classificationcited by 1
  16. 2026Benchmarking Large Language Models for ABAP Code Generation: An Empirical Study on Iterative Improvement by Compiler Feedbackcited by 1
  17. 2024Contextual Augmented Multi-Model Programming (CAMP): A Hybrid Local-Cloud Copilot Frameworkcited by 0
  18. 2025Security smells in infrastructure as code: a taxonomy update beyond the seven sinscited by 0
  19. 2025Selecting and Combining Large Language Models for Scalable Code Clone Detectioncited by 0
  20. 2026Nemobot Games: Crafting Strategic AI Gaming Agents for Interactive Learning with Large Language Modelscited by 0