mlsanityData to 7 Oct 2026

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

arXiv 2605.25536 ↗First posted 25 May 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 102 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

35 of 102 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
23 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 264
  3. 2020Unsupervised Translation of Programming Languagescited by 262
  4. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  5. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  6. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
  7. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  8. 2023Towards an Understanding of Large Language Models in Software Engineering Taskscited by 67
  9. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 65
  10. 2023A Survey on Large Language Models for Software Engineeringcited by 58
  11. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  12. 2023Automated Paper Screening for Clinical Reviews Using Large Language Modelscited by 29
  13. 2025Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Taskscited by 17
  14. 2024The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviewscited by 14
  15. 2023A Survey on Automated Program Repair Techniquescited by 12
  16. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  17. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
  18. 2020Systematic Literature Reviews in Software Engineering -- Enhancement of the Study Selection Process using Cohen's Kappa Statisticcited by 7
  19. 2022Machine Learning for Software Engineering: A Tertiary Studycited by 7
  20. 2023How many preprints have actually been printed and why: a case study of computer science preprints on arXivcited by 6
  21. 2014A Critical Review of "Automatic Patch Generation Learned from Human-Written Patches": Essay on the Problem Statement and the Evaluation of Automatic Software Repaircited by 4
  22. 2023Using a Nearest-Neighbour, BERT-Based Approach for Scalable Clone Detectioncited by 4
  23. 2024Using Ensemble Inference to Improve Recall of Clone Detectioncited by 2

Built on by

Not cited yet.