mlsanityData to 14 Oct 2026

LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models

arXiv 2505.19240 ↗First posted 25 May 2025cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) research has grown rapidly, along with increasing concern about their limitations. In this survey, we conduct a data-driven, semi-automated review of research on limitations of LLMs (LLLMs) from 2022 to early 2025 using a bottom-up approach. From a corpus of 250,000 ACL and arXiv papers, we identify 14,648 relevant papers using keyword filtering, LLM-based classification, validated against expert labels, and topic clustering (via two approaches, HDBSCAN+BERTopic and LlooM). We find that the share of LLM-related papers increases over fivefold in ACL and nearly eightfold in arXiv between 2022 and 2025. Since 2022, LLLMs research grows even faster, reaching over 30% of LLM papers by 2025. Reasoning remains the most studied limitation, followed by generalization, hallucination, bias, and security. The distribution of topics in the ACL dataset stays relatively stable over time, while arXiv shifts toward security risks, alignment, hallucinations, knowledge editing, and multimodality. We offer a quantitative view of trends in LLLMs research and release a dataset of annotated abstracts and a validated methodology, available at: https://github.com/a-kostikova/LLLMs-Survey.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 64 of its 94 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

64 of 94 references.

References · 64

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
38 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  2. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  3. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  4. 2023A Comprehensive Overview of Large Language Modelscited by 648
  5. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  8. 2023A Survey on Large Language Models for Recommendationcited by 379
  9. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  10. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  11. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  12. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  13. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  14. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  15. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  16. 2023Large Language Model Alignment: A Surveycited by 243
  17. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  18. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  19. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  20. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  21. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  22. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
  23. 2025LLM4SR: A Survey on Large Language Models for Scientific Researchcited by 83
  24. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  25. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
  26. 2024LLMs Will Always Hallucinate, and We Need to Live With Thiscited by 78
  27. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  28. 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 74
  29. 2023Large Language Models Enable Few-Shot Clusteringcited by 72
  30. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 70
  31. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  32. 2025PaSa: An LLM Agent for Comprehensive Academic Paper Searchcited by 64
  33. 2024LitLLM: A Toolkit for Scientific Literature Reviewcited by 63
  34. 2023Red Teaming Language Model Detectors with Language Modelscited by 55
  35. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 51
  36. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitationscited by 42
  37. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  38. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 35

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesiscited by 3
  2. 2025Multi-dimensional Data Analysis and Applications Basing on LLM Agents and Knowledge Graph Interactionscited by 1
  3. 2025StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systemscited by 1
  4. 2026NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpuscited by 1
  5. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1
  6. 2025Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGIcited by 0
  7. 2026Multi-Dimensional Knowledge Profiling with Large-Scale Literature Database and Hierarchical Retrievalcited by 0
  8. 2026X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probescited by 0
  9. 2026RAGe: A Retrieval-Augmented Generation Evaluation Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Large Language Models (Really) Need Statistical Foundations?
  2. 2024Multilingual Large Language Models: A Systematic Survey
  3. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  4. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  5. 2024Awes, Laws, and Flaws From Today's LLM Research
  6. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
  7. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
  8. 2025On the Fundamental Limits of LLMs at Scale