# Can Language Models Act as Knowledge Bases at Scale?

arXiv: https://arxiv.org/abs/2402.14273 · HTML: https://mlsanity.com/paper/2402.14273/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 30 of 33)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](https://mlsanity.com/paper/2201.08860/) (2022)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [Variance Reduction in SGD by Distributed Importance Sampling](https://mlsanity.com/paper/1511.06481/) (2015)
- [Open-World Knowledge Graph Completion](https://mlsanity.com/paper/1711.03438/) (2017)
- [Privacy Issues in Large Language Models: A Survey](https://mlsanity.com/paper/2312.06717/) (2023)
- [Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](https://mlsanity.com/paper/2007.00849/) (2020)
- [Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation](https://mlsanity.com/paper/2112.07924/) (2021)
- [Evaluating Language Models for Knowledge Base Completion](https://mlsanity.com/paper/2303.11082/) (2023)
- [Rethinking Knowledge Graph Evaluation Under the Open-World Assumption](https://mlsanity.com/paper/2209.08858/) (2022)

## Built on by (most cited first, 17 of 17)

- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Do Large Language Models Know How Much They Know?](https://mlsanity.com/paper/2502.19573/) (2025)
- [How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency](https://mlsanity.com/paper/2407.13578/) (2024)
- [Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training](https://mlsanity.com/paper/2502.04066/) (2025)
- [Multimodal Contrastive Representation Learning in Augmented Biomedical Knowledge Graphs](https://mlsanity.com/paper/2501.01644/) (2025)
- [Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledge](https://mlsanity.com/paper/2506.06240/) (2025)
- [GoalLadder: Incremental Goal Discovery with Vision-Language Models](https://mlsanity.com/paper/2506.16396/) (2025)
- [60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering](https://mlsanity.com/paper/2409.15825/) (2024)
- [Are LLMs Really Not Knowledgeable? Mining the Submerged Knowledge in LLMs' Memory](https://mlsanity.com/paper/2412.20846/) (2024)
- [Benchmarking Large Language Models for Knowledge Graph Validation](https://mlsanity.com/paper/2602.10748/) (2026)
- [On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classification](https://mlsanity.com/paper/2408.09585/) (2024)
- [Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing](https://mlsanity.com/paper/2408.14849/) (2024)
- [Latent Feature Mining for Predictive Model Enhancement with Large Language Models](https://mlsanity.com/paper/2410.04347/) (2024)
- [MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset](https://mlsanity.com/paper/2412.10105/) (2024)
- [A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models](https://mlsanity.com/paper/2505.19286/) (2025)
- [How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models](https://mlsanity.com/paper/2509.19371/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable proficiency in understanding and generating responses to complex queries through large-scale pre-training. However, the efficacy of these models in memorizing and reasoning among large-scale structured knowledge, especially world knowledge that explicitly covers abundant factual information remains questionable. Addressing this gap, our research investigates whether LLMs can effectively store, recall, and reason with knowledge on a large scale comparable to latest knowledge bases (KBs) such as Wikidata. Specifically, we focus on three crucial aspects to study the viability: (1) the efficiency of LLMs with different sizes in memorizing the exact knowledge in the large-scale KB; (2) the flexibility of recalling the memorized knowledge in response to natural language queries; (3) the capability to infer new knowledge through reasoning. Our findings indicate that while LLMs hold promise as large-scale KBs capable of retrieving and responding with flexibility, enhancements in their reasoning capabilities are necessary to fully realize their potential.