mlsanityData to 7 Oct 2026

Automatic Programming: Large Language Models and Beyond

arXiv 2405.02213 ↗First posted 3 May 2024cs.SE cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Automatic programming has seen increasing popularity due to the emergence of tools like GitHub Copilot which rely on Large Language Models (LLMs). At the same time, automatically generated code faces challenges during deployment due to concerns around quality and trust. In this article, we study automated coding in a general sense and study the concerns around code quality, security and related issues of programmer responsibility. These are key issues for organizations while deciding on the usage of automatically generated code. We discuss how advances in software engineering such as program repair and analysis can enable automatic programming. We conclude with a forward looking view, focusing on the programming environment of the near future, where programmers may need to switch to different roles to fully utilize the power of automatic programming. Automated repair of automatically generated programs from LLMs, can help produce higher assurance code from LLMs, along with evidence of assurance

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 81 of its 160 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

81 of 160 references.

References · 81

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
38 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  5. 2023Textbooks Are All You Needcited by 561
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  7. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  8. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  9. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  10. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  11. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  12. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  13. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  14. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  15. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  16. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  17. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  18. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  19. 2018Mapping Language to Code in Programmatic Contextcited by 104
  20. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  21. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 95
  22. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  23. 2018TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generationcited by 88
  24. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  25. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 87
  26. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  27. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 81
  28. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 78
  29. 2016Latent Predictor Networks for Code Generationcited by 73
  30. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  31. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  32. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
  33. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 63
  34. 2019TreeGen: A Tree-Based Transformer Architecture for Code Generationcited by 58
  35. 2021A Syntax-Guided Edit Decoder for Neural Program Repaircited by 57
  36. 2018Tools and Benchmarks for Automated Log Parsingcited by 56
  37. 2021Towards Automating Code Review Activitiescited by 54
  38. 2021Neural Program Repair with Execution-based Backpropagationcited by 50

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 17; 2026: 13.

35 papers.

Sort:

By area:

  1. 2024SpecRover: Code Intent Extraction via LLMscited by 71
  2. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHubcited by 53
  3. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Modelscited by 49
  4. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  5. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  6. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  7. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  8. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  9. 2025AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generationcited by 8
  10. 2025Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overviewcited by 7
  11. 2025Evaluation of LLMs for mathematical problem solvingcited by 7
  12. 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agentscited by 6
23 more
  1. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 4
  2. 2025Thinking Machines: Mathematical Reasoning in the Age of LLMscited by 4
  3. 2025AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Softwarecited by 4
  4. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  5. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  6. 2026Why Human Guidance Matters in Collaborative Vibe Codingcited by 3
  7. 2024Software Engineering for Collective Cyber-Physical Ecosystemscited by 2
  8. 2025LLM Company Policies and Policy Implications in Software Organizationscited by 2
  9. 2025TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Frameworkcited by 1
  10. 2025A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AIcited by 1
  11. 2026AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptationcited by 1
  12. 2026From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineeringcited by 1
  13. 2024WarriorCoder: Learning from Expert Battles to Augment Code Large Language Modelscited by 0
  14. 2025Will AI replace Software Engineers? Do not hold your breathcited by 0
  15. 2025What Makes Code Generation Ethically Sourced?cited by 0
  16. 2026Prompt Driven Development with Claude Code: Building a Complete TUI Framework for the Ring Programming Languagecited by 0
  17. 2026Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safetycited by 0
  18. 2026CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generationcited by 0
  19. 2026From Code-Centric to Intent-Centric Software Engineering: A Reflexive Thematic Analysis of Generative AI, Agentic Systems, and Engineering Accountabilitycited by 0
  20. 2026Projectional Decoding: Towards Semantic-Aware LLM Generationcited by 0
  21. 2026Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Codecited by 0
  22. 2026Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inferencecited by 0
  23. 2026The Vibe Shift in Software Engineering: Evaluating AI-Led Conversational Programming for Performance, Cognition, and Responsible Adoptioncited by 0