# Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5

arXiv: https://arxiv.org/abs/2510.07024 · HTML: https://mlsanity.com/paper/2510.07024/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomy](https://mlsanity.com/paper/2308.11884/) (2023)
- [Enabling LLM Knowledge Analysis via Extensive Materialization](https://mlsanity.com/paper/2411.04920/) (2024)
- [Evaluating the Knowledge Base Completion Potential of GPT](https://mlsanity.com/paper/2310.14771/) (2023)
- [Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extraction](https://mlsanity.com/paper/2404.12957/) (2024)
- [GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledge](https://mlsanity.com/paper/2507.05740/) (2025)
- [Exploring Wikipedia Gender Diversity Over Time unicodex2013 The Wikipedia Gender Dashboard (WGD)](https://mlsanity.com/paper/2501.12610/) (2025)

## Built on by (most cited first, 3 of 3)

- [LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale](https://mlsanity.com/paper/2603.24080/) (2026)
- [LLM-Metrics: Measuring Research Impact Through Large Language Model Memory](https://mlsanity.com/paper/2605.22176/) (2026)
- [Constructing Disambiguated Knowledge Bases from Large Language Models at Scale](https://mlsanity.com/paper/2608.03729/) (2026)

## Abstract (arXiv)

LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (Hu et al., 2025a), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion benchmarks, which are prone to availability bias. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.