# How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency

arXiv: https://arxiv.org/abs/2407.13578 · HTML: https://mlsanity.com/paper/2407.13578/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Factuality of Large Language Models: A Survey](https://mlsanity.com/paper/2402.02420/) (2024)
- [TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness](https://mlsanity.com/paper/2402.12545/) (2024)
- [Why Does New Knowledge Create Messy Ripple Effects in LLMs?](https://mlsanity.com/paper/2407.12828/) (2024)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2404.00942/) (2024)
- [Evaluating Consistency and Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2404.16478/) (2024)

## Built on by (most cited first, 10 of 10)

- [Grounding LLM Reasoning with Knowledge Graphs](https://mlsanity.com/paper/2502.13247/) (2025)
- [Neuro-Symbolic Frameworks: Conceptual Characterization and Empirical Comparative Analysis](https://mlsanity.com/paper/2509.07122/) (2025)
- [DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents](https://mlsanity.com/paper/2608.18524/) (2026)
- [A systematic review of relation extraction task since the emergence of Transformers](https://mlsanity.com/paper/2511.03610/) (2025)
- [Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration](https://mlsanity.com/paper/2601.18901/) (2026)
- [Verification of the Implicit World Model in a Generative Model via Adversarial Sequences](https://mlsanity.com/paper/2602.05903/) (2026)
- [Benchmarking Large Language Models for Knowledge Graph Validation](https://mlsanity.com/paper/2602.10748/) (2026)
- [How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models](https://mlsanity.com/paper/2509.19371/) (2025)
- [From Storage to Access: Verifiable Activation of Parametric Knowledge in LLMs via Explicit Priming and Implicit Reasoning](https://mlsanity.com/paper/2608.18581/) (2026)
- [NeuroGraph: An AI Graph-Driven Neuro-Symbolic Framework for Explainable Threat Reasoning in Advanced Manufacturing](https://mlsanity.com/paper/2609.00604/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs), yet current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance. In this work, we rethink the requirements for evaluating reliable LLM-as-KB usage and highlight two essential factors: factuality, ensuring accurate responses to seen and unseen knowledge, and consistency, maintaining stable answers to questions about the same knowledge. We introduce UnseenQA, a dataset designed to assess LLM performance on unseen knowledge, and propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score. Our experiments on 26 LLMs reveal several challenges regarding their use as KBs, underscoring the need for more principled and comprehensive evaluation.