mlsanityData to 14 Oct 2026

How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency

arXiv 2407.13578 ↗First posted 18 Jul 2024cs.CL cs.AIcited by 10

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs), yet current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance. In this work, we rethink the requirements for evaluating reliable LLM-as-KB usage and highlight two essential factors: factuality, ensuring accurate responses to seen and unseen knowledge, and consistency, maintaining stable answers to questions about the same knowledge. We introduce UnseenQA, a dataset designed to assess LLM performance on unseen knowledge, and propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score. Our experiments on 26 LLMs reveal several challenges regarding their use as KBs, underscoring the need for more principled and comprehensive evaluation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 37 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

11 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Evaluating Consistency and Reasoning Capabilities of Large Language Models25 Apr
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 9

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  4. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 168
  5. 2024Factuality of Large Language Models: A Surveycited by 66
  6. 2024TrustScore: Reference-Free Evaluation of LLM Response Trustworthinesscited by 19
  7. 2024Why Does New Knowledge Create Messy Ripple Effects in LLMs?cited by 18
  8. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  9. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Grounding LLM Reasoning with Knowledge Graphscited by 7
  2. 2025Neuro-Symbolic Frameworks: Conceptual Characterization and Empirical Comparative Analysiscited by 2
  3. 2026DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agentscited by 2
  4. 2025A systematic review of relation extraction task since the emergence of Transformerscited by 1
  5. 2026Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibrationcited by 1
  6. 2026Verification of the Implicit World Model in a Generative Model via Adversarial Sequencescited by 1
  7. 2026Benchmarking Large Language Models for Knowledge Graph Validationcited by 1
  8. 2025How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Modelscited by 0
  9. 2026From Storage to Access: Verifiable Activation of Parametric Knowledge in LLMs via Explicit Priming and Implicit Reasoningcited by 0
  10. 2026NeuroGraph: An AI Graph-Driven Neuro-Symbolic Framework for Explainable Threat Reasoning in Advanced Manufacturingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
  2. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  3. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
  4. 2025Inside-Out: Hidden Factual Knowledge in LLMs
  5. 2025SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs
  6. 2024Logical Consistency of Large Language Models in Fact-checking
  7. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  8. 2024LLMs' Understanding of Natural Language Revealed