# LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale

arXiv: https://arxiv.org/abs/2603.24080 · HTML: https://mlsanity.com/paper/2603.24080/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Builds on (most cited first, 21 of 21)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations](https://mlsanity.com/paper/2503.04761/) (2025)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [Enabling LLM Knowledge Analysis via Extensive Materialization](https://mlsanity.com/paper/2411.04920/) (2024)
- [Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts](https://mlsanity.com/paper/2308.10410/) (2023)
- [VeriFastScore: Speeding up long-form factuality evaluation](https://mlsanity.com/paper/2505.16973/) (2025)
- [GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledge](https://mlsanity.com/paper/2507.05740/) (2025)
- [Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness](https://mlsanity.com/paper/2510.06780/) (2025)
- [Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5](https://mlsanity.com/paper/2510.07024/) (2025)
- [Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs](https://mlsanity.com/paper/2511.01187/) (2025)
- [How Similar Are Grokipedia and Wikipedia? A Multi-Dimensional Textual and Structural Comparison](https://mlsanity.com/paper/2510.26899/) (2025)

## Built on by (most cited first, 3 of 3)

- [Beyond Questions: Evaluating the Stated Knowledge of LLMs](https://mlsanity.com/paper/2605.26937/) (2026)
- [Constructing Disambiguated Knowledge Bases from Large Language Models at Scale](https://mlsanity.com/paper/2608.03729/) (2026)
- [LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs](https://mlsanity.com/paper/2609.01182/) (2026)

## Abstract (arXiv)

Benchmarks like MMLU suggest flagship language models approach factuality saturation above 90%. LLMpedia shows this picture is incomplete. We materialize ~1.3M encyclopedia articles entirely from parametric memory across three model families, then audit every claim against Wikipedia and curated web evidence. For textttgpt-5-mini, the verifiable true rate is 68.4% on Wikipedia-covered subjects - more than 21,pp below MMLU - and the gap is driven by unverifiability (30.5%), not refutation (1.2%). Beyond Wikipedia, frontier articles audited against curated web evidence reach 57.6%; Wikipedia covers only 56.7% of model-surfaced subjects, and three model families overlap in just 7.3% of subject choices. In a retrieval-trap benchmark inspired by prior analysis of Grokipedia, LLMpedia is more factual at roughly half the textual similarity to Wikipedia. Every prompt, article, and verdict is released. Data, code, interface: https://llmpedia.net.