mlsanityData to 7 Oct 2026

Large Language Models: A Survey

arXiv 2402.06196 ↗First posted 9 Feb 2024cs.CL cs.AIcited by 506

Not on the map. Why?

Cited by 61 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws citekaplan2020scaling,hoffmann2022training. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Key facts. Cited by 506 papers in the map (data through 7 Oct 2026). Builds on 166 of its 241 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

166 of 241 references.

References · 166

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  12. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
38 more
  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  13. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  14. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  15. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  16. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  17. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  18. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  19. 2017Deep reinforcement learning from human preferencescited by 3,846
  20. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  21. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  22. 2021Program Synthesis with Large Language Modelscited by 3,509
  23. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  24. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  25. 2020Longformer: The Long-Document Transformercited by 3,362
  26. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  27. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  28. 2023Mistral 7Bcited by 3,065
  29. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  30. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  31. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  32. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  33. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  34. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  35. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  36. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  37. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  38. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674

50 most cited of 166 All 166 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 152; 2025: 226; 2026: 128.

506 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2025Memory in the Age of AI Agentscited by 285
  4. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 234
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  6. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  8. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  9. 2025Agentic Large Language Models, a surveycited by 105
  10. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  11. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  12. 2024Recent Advances in Speech Language Models: A Surveycited by 85
38 more
  1. 2024Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localizationcited by 81
  2. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  3. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  4. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  5. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  6. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  7. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  8. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  9. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  10. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  11. 2024RATT: A Thought Structure for Coherent and Correct LLM Reasoningcited by 39
  12. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  13. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  14. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 38
  15. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  16. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  17. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 35
  18. 2025A Survey of Direct Preference Optimizationcited by 35
  19. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  20. 2025CARROT: A Cost Aware Rate Optimal Routercited by 34
  21. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  22. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 32
  23. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  24. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  25. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 29
  26. 2025Large Language Models for Multi-Robot Systems: A Surveycited by 26
  27. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  28. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 26
  29. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  30. 2025A Survey on Human Interaction Motion Generationcited by 25
  31. 2025PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaborationcited by 25
  32. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  33. 2024Causality for Large Language Modelscited by 24
  34. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
  35. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
  36. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  37. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  38. 2025Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularitycited by 22

50 most cited of 506 All 506 (JSON)