# Measuring and Modifying Factual Knowledge in Large Language Models

arXiv: https://arxiv.org/abs/2306.06264 · HTML: https://mlsanity.com/paper/2306.06264/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 24 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings](https://mlsanity.com/paper/2203.00211/) (2022)

## Built on by (most cited first, 16 of 16)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [Distinguishing the Knowable from the Unknowable with Language Models](https://mlsanity.com/paper/2402.03563/) (2024)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning](https://mlsanity.com/paper/2504.06438/) (2025)
- [Identifying Query-Relevant Neurons in Large Language Models for Long-Form Texts](https://mlsanity.com/paper/2406.10868/) (2024)
- [Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA](https://mlsanity.com/paper/2412.10079/) (2024)
- [FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning](https://mlsanity.com/paper/2410.06304/) (2024)
- [ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models](https://mlsanity.com/paper/2403.05266/) (2024)
- [Existing Large Language Model Unlearning Evaluations Are Inconclusive](https://mlsanity.com/paper/2506.00688/) (2025)
- [Operationalising the Superficial Alignment Hypothesis via Task Complexity](https://mlsanity.com/paper/2602.15829/) (2026)
- [Language Models as Knowledge Bases for Visual Word Sense Disambiguation](https://mlsanity.com/paper/2310.01960/) (2023)
- [Evaluating LLMs for Gender Disparities in Notable Persons](https://mlsanity.com/paper/2403.09148/) (2024)
- [MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Models](https://mlsanity.com/paper/2512.21231/) (2025)
- [Knowledge Boundary Discovery for Large Language Models](https://mlsanity.com/paper/2603.21022/) (2026)
- [On the Analogy between Human Brain and LLMs: Spotting Key Neurons in Grammar Perception](https://mlsanity.com/paper/2511.06519/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) store an extensive amount of factual knowledge obtained from vast collections of text. To effectively utilize these models for downstream tasks, it is crucial to have reliable methods for measuring their knowledge. However, existing approaches for knowledge measurement have certain limitations, and despite recent efforts, they fail to provide accurate measurements and the necessary insights for modifying the knowledge within LLMs. In this work, we employ information theory-based measurements to provide a framework estimating the factual knowledge contained within large language models. More specifically, we measure knowledge by analyzing the LLM's prediction probability distribution before and after instilling the target knowledge, employing metrics such as entropy and KL-divergence. Introducing our metrics, we first assess their accuracy in comparison to previous ranking-based methods, surpassing them by over 35% in a synthetic experiment. Then, we explore two prominent methods of knowledge instillation, discovering that LLMs exhibit limitations in capturing new knowledge under specific circumstances for one of these methods. Lastly, we demonstrate the applicability of our methods in extracting unlearned and mislearned facts in LLMs through their application to in-context learning. We make code and data for all methods and experiments in this paper publicly available.