mlsanityData to 14 Oct 2026

Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code

arXiv 2311.07989 ↗First posted 14 Nov 2023cs.CL cs.AI cs.SEcited by 62

Not on the map. Why?

Abstract (arXiv)

In this work we systematically review the recent advancements in software engineering with language models, covering 70+ models, 40+ evaluation tasks, 180+ datasets, and 900 related works. Unlike previous works, we integrate software engineering (SE) with natural language processing (NLP) by discussing the perspectives of both sides: SE applies language models for development automation, while NLP adopts SE tasks for language model evaluation. We break down code processing models into general language models represented by the GPT family and specialized models that are specifically pretrained on code, often with tailored objectives. We discuss the relations and differences between these models, and highlight the historical transition of code modeling from statistical models and RNNs to pretrained Transformers and LLMs, which is exactly the same course that had been taken by NLP. We also go beyond programming and review LLMs' application in other software engineering activities including requirement engineering, testing, deployment, and operations in an endeavor to provide a global view of NLP in SE, and identify key challenges and potential future directions in this domain. We keep the survey open and updated on GitHub at https://github.com/codefuse-ai/Awesome-Code-LLM.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 203 of its 902 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

203 of 902 references.

References · 203

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2016Layer Normalizationcited by 5,669
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
38 more
  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Mistral 7Bcited by 3,164
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  14. 2023PaLM 2 Technical Reportcited by 1,162
  15. 2023Teaching Large Language Models to Self-Debugcited by 988
  16. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  17. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  18. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  19. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  20. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  21. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  22. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  23. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  24. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  25. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  26. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  27. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  28. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  29. 2023Textbooks Are All You Needcited by 569
  30. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  31. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  32. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  33. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  34. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  35. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  36. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  37. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  38. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333

50 most cited of 203 All 203 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 34; 2025: 19; 2026: 8.

62 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  3. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  4. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 134
  5. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 93
  6. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
  7. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
  8. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  9. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  10. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  11. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  12. 2024Multiple-Choice Questions are Efficient and Robust LLM Evaluatorscited by 28
38 more
  1. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  2. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  3. 2024BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Taskscited by 19
  4. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  5. 2024A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoningcited by 15
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  7. 2025MaintainCoder: Maintainable Code Generation Under Dynamic Requirementscited by 14
  8. 2025ReCode: Updating Code API Knowledge with Reinforcement Learningcited by 12
  9. 2024DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuningcited by 11
  10. 2024Your Context Is Not an Array: Unveiling Random Access Limitations in Transformerscited by 11
  11. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 11
  12. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  13. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  14. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 9
  15. 2024Natural Language Outlines for Code: Literate Programming in the LLM Eracited by 9
  16. 2024CodeV: Issue Resolving with Visual Datacited by 9
  17. 2025Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latencycited by 9
  18. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generationcited by 8
  19. 2024Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketchescited by 7
  20. 2024GALLa: Graph Aligned Large Language Models for Improved Source Code Understandingcited by 7
  21. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  22. 2026What makes a harness a harness: necessary and sufficient conditions for an agent harnesscited by 6
  23. 2024The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelinescited by 5
  24. 2024Understanding Defects in Generated Codes by Language Modelscited by 5
  25. 2025MERA Code: A Unified Framework for Evaluating Code Generation Across Taskscited by 4
  26. 2025Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguitycited by 4
  27. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  28. 2025Skill Discovery for Software Scripting Automation via Offline Simulations with LLMscited by 3
  29. 2025C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Poolingcited by 3
  30. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  31. 2026Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answeringcited by 3
  32. 2024Test Code Generation for Telecom Software Systems using Two-Stage Generative Modelcited by 2
  33. 2024DocuMint: Docstring Generation for Python using Small Language Modelscited by 2
  34. 2024Exploring the Integration of Large Language Models in Industrial Test Maintenance Processescited by 2
  35. 2025CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMscited by 2
  36. 2025Evaluating Software Process Models for Multi-Agent Class-Level Code Generationcited by 2
  37. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputscited by 2
  38. 2026SWE-QA: A Dataset and Benchmark for Complex Code Understandingcited by 2

50 most cited of 62 All 62 (JSON)