mlsanityData to 14 Oct 2026

A Survey of Large Language Models

arXiv 2303.18223 ↗First posted 31 Mar 2023cs.CL cs.AIcited by 2,576

Not on the map. Why?

Cited by 349 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

Key facts. Cited by 2,576 papers in the map (data through 14 Oct 2026). Builds on 319 of its 461 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

319 of 461 references.

References · 319

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2017Proximal Policy Optimization Algorithmscited by 15,215
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  12. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
38 more
  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2016Layer Normalizationcited by 5,669
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  13. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  14. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  15. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  16. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  17. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  18. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  19. 2017Deep reinforcement learning from human preferencescited by 3,912
  20. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  21. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  22. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  23. 2021Program Synthesis with Large Language Modelscited by 3,637
  24. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  25. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  26. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  27. 2016Pointer Sentinel Mixture Modelscited by 3,096
  28. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  29. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  30. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  31. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  32. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  33. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  34. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  35. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  36. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  37. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  38. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592

50 most cited of 319 All 319 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 449; 2024: 870; 2025: 818; 2026: 439.

2,576 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  7. 2025Large Language Diffusion Modelscited by 912
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  10. 2024Chronos: Learning the Language of Time Seriescited by 759
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  4. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  6. 2024Large Language Models: A Surveycited by 525
  7. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  9. 2024Understanding the planning of LLM agents: A surveycited by 421
  10. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  11. 2023Scaling Data-Constrained Language Modelscited by 403
  12. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  13. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  14. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  15. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  16. 2023A Survey on Large Language Models for Recommendationcited by 379
  17. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  18. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  19. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  20. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  21. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  22. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  23. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  24. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  25. 2024Discrete Flow Matchingcited by 317
  26. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  27. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  28. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  29. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 276
  30. 2023A Survey on Model Compression for Large Language Modelscited by 269
  31. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  32. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  33. 2023Challenges and Applications of Large Language Modelscited by 254
  34. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  35. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  36. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  37. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  38. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243

50 most cited of 2,576 All 2,576 (JSON)