# Tracing Relational Knowledge Recall in Large Language Models

arXiv: https://arxiv.org/abs/2604.19934 · HTML: https://mlsanity.com/paper/2604.19934/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs](https://mlsanity.com/paper/1801.05453/) (2018)
- [Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs](https://mlsanity.com/paper/2402.07321/) (2024)
- [Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models](https://mlsanity.com/paper/2502.10835/) (2025)
- [Embedded Named Entity Recognition using Probing Classifiers](https://mlsanity.com/paper/2403.11747/) (2024)
- [On Entity Identification in Language Models](https://mlsanity.com/paper/2506.02701/) (2025)

## Built on by (most cited first, 2 of 2)

- [Relational Linear Properties in Language Models: An Empirical Investigation](https://mlsanity.com/paper/2605.22532/) (2026)
- [Relation Before Entity: Deferred Commitment in Language Model Factual Recall](https://mlsanity.com/paper/2609.17537/) (2026)

## Abstract (arXiv)

We study how large language models recall relational knowledge during text generation, with a focus on identifying latent representations suitable for relation classification via linear probes. Prior work shows how attention heads and MLPs interact to resolve subject, predicate, and object, but it remains unclear which representations support faithful linear relation classification and why some relation types are easier to capture linearly than others. We systematically evaluate different latent representations derived from attention head and MLP contributions, showing that per-head attention contributions to the residual stream are comparatively strong features for linear relation classification. Feature attribution analyses of the trained probes, as well as characteristics of the different relation types, reveal clear correlations between probe accuracy and relation specificity, entity connectedness, and how distributed the signal on which the probe relies is across attention heads. Finally, we show how token-level feature attribution of probe predictions can be used to reveal probe behavior in further detail.