mlsanityData to 14 Oct 2026

Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?

arXiv 2308.10168 ↗First posted 20 Aug 2023cs.CLcited by 168

Not on the map. Why?

Cited by 27 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Since the recent prosperity of Large Language Models (LLMs), there have been interleaved discussions regarding how to reduce hallucinations from LLM responses, how to increase the factuality of LLMs, and whether Knowledge Graphs (KGs), which store the world knowledge in a symbolic form, will be replaced with LLMs. In this paper, we try to answer these questions from a new angle: How knowledgeable are LLMs? To answer this question, we constructed Head-to-Tail, a benchmark that consists of 18K question-answer (QA) pairs regarding head, torso, and tail facts in terms of popularity. We designed an automated evaluation method and a set of metrics that closely approximate the knowledge an LLM confidently internalizes. Through a comprehensive evaluation of 16 publicly available LLMs, we show that existing LLMs are still far from being perfect in terms of their grasp of factual knowledge, especially for facts of torso-to-tail entities.

Key facts. Cited by 168 papers in the map (data through 14 Oct 2026). Builds on 23 of its 48 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

23 of 48 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  8. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  12. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
11 more
  1. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  2. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  3. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  4. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  5. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  6. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  7. 2020KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoningcited by 106
  8. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  9. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  10. 2023An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answeringcited by 19
  11. 2023Knowledge Base Completion for Long-Tail Entitiescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 63; 2025: 50; 2026: 31.

168 papers.

Sort:

By area:

  1. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  2. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  3. 2025Why Language Models Hallucinatecited by 220
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  5. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  6. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  7. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 101
  8. 2024CRAG -- Comprehensive RAG Benchmarkcited by 97
  9. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  10. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  11. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  12. 2024STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Basescited by 55
38 more
  1. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  2. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 50
  3. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 50
  4. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
  5. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 45
  6. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  7. 2023Ever: Mitigating Hallucination in Large Language Models through Real-Time Verification and Rectificationcited by 38
  8. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  9. 2024KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detectioncited by 33
  10. 2024Calibrating Long-form Generations from Large Language Modelscited by 32
  11. 2023Emptying the Ocean with a Spoon: Should We Edit Models?cited by 27
  12. 2023Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologiescited by 25
  13. 2024DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMscited by 24
  14. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 24
  15. 2024REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Modelscited by 23
  16. 2025AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corporacited by 23
  17. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledgecited by 20
  18. 2024Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Modelscited by 19
  19. 2024Are Large Language Models a Good Replacement of Taxonomies?cited by 19
  20. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  21. 2024Contextualization Distillation from Large Language Model for Knowledge Graph Completioncited by 18
  22. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  23. 2023AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graphcited by 17
  24. 2024A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applicationscited by 17
  25. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 17
  26. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  27. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 16
  28. 2024Exploring the Potential of Large Language Models in Graph Generationcited by 15
  29. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  30. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 15
  31. 2024ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Modelscited by 14
  32. 2024RAG based Question-Answering for Contextual Response Prediction Systemcited by 14
  33. 2024VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understandingcited by 13
  34. 2025How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?cited by 13
  35. 2025Learning Facts at Scale with Active Readingcited by 13
  36. 2024OneEdit: A Neural-Symbolic Collaboratively Knowledge Editing Systemcited by 12
  37. 2024Enabling LLM Knowledge Analysis via Extensive Materializationcited by 12
  38. 2025Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learningcited by 12

50 most cited of 168 All 168 (JSON)