mlsanityData to 7 Oct 2026

Code LLMs: A Taxonomy-based Survey

arXiv 2412.08291 ↗First posted 11 Dec 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks and have recently expanded their impact to coding tasks, bridging the gap between natural languages (NL) and programming languages (PL). This taxonomy-based survey provides a comprehensive analysis of LLMs in the NL-PL domain, investigating how these models are utilized in coding tasks and examining their methodologies, architectures, and training processes. We propose a taxonomy-based framework that categorizes relevant concepts, providing a unified classification system to facilitate a deeper understanding of this rapidly evolving field. This survey offers insights into the current state and future directions of LLMs in coding tasks, including their applications and limitations.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 61 of its 86 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

61 of 86 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2024Mixtral of Expertscited by 1,785
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  13. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  14. 2023StarCoder: may the source be with you!cited by 790
  15. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  16. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  17. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  18. 2016Language Modeling with Gated Convolutional Networkscited by 688
  19. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  20. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  21. 2023Textbooks Are All You Needcited by 561
  22. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  23. 2023The Falcon Series of Open Language Modelscited by 526
  24. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  25. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  26. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  27. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  28. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  29. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  30. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  31. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  32. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  33. 2023SantaCoder: don't reach for the stars!cited by 169
  34. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  35. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  36. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  37. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  38. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025TigerLLM - A Family of Bangla Large Language Modelscited by 15
  2. 2026CodeGuard: Improving LLM Guardrails in CS Educationcited by 1
  3. 2026A Taxonomy of Programming Languages for Code Generationcited by 0