# A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models

arXiv: https://arxiv.org/abs/2505.19286 · HTML: https://mlsanity.com/paper/2505.19286/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 23 of 23)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Is Homophily a Necessity for Graph Neural Networks?](https://mlsanity.com/paper/2106.06134/) (2021)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [CoDEx: A Comprehensive Knowledge Graph Completion Benchmark](https://mlsanity.com/paper/2009.07810/) (2020)
- [Message Passing for Hyper-Relational Knowledge Graphs](https://mlsanity.com/paper/2009.10847/) (2020)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [Distinguishing the Knowable from the Unknowable with Language Models](https://mlsanity.com/paper/2402.03563/) (2024)
- [Tree Decomposed Graph Neural Network](https://mlsanity.com/paper/2108.11022/) (2021)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Triple Trustworthiness Measurement for Knowledge Graph](https://mlsanity.com/paper/1809.09414/) (2018)
- [Rethinking Language Models as Symbolic Knowledge Graphs](https://mlsanity.com/paper/2308.13676/) (2023)
- [KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions](https://mlsanity.com/paper/2407.05868/) (2024)
- [Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Base](https://mlsanity.com/paper/2503.23361/) (2025)
- [KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphs](https://mlsanity.com/paper/2312.11539/) (2023)

## Abstract (arXiv)

Large language models have been extensively studied as neural knowledge bases for their knowledge access, editability, reasoning, and explainability. However, few works focus on the structural patterns of their knowledge. Motivated by this gap, we investigate these structural patterns from a graph perspective. We quantify the knowledge of LLMs at both the triplet and entity levels, and analyze how it relates to graph structural properties such as node degree. Furthermore, we uncover the knowledge homophily, where topologically close entities exhibit similar levels of knowledgeability, which further motivates us to develop graph machine learning models to estimate entity knowledge based on its local neighbors. This model further enables valuable knowledge checking by selecting triplets less known to LLMs. Empirical results show that using selected triplets for fine-tuning leads to superior performance.