# Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs

arXiv: https://arxiv.org/abs/2402.07321 · HTML: https://mlsanity.com/paper/2402.07321/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 11 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Inducing Causal Structure for Interpretable Neural Networks](https://mlsanity.com/paper/2112.00826/) (2021)
- [An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4L](https://mlsanity.com/paper/2310.07325/) (2023)

## Built on by (most cited first, 30 of 42)

- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Knowledge Circuits in Pretrained Transformers](https://mlsanity.com/paper/2405.17969/) (2024)
- [Interpreting Attention Layer Outputs with Sparse Autoencoders](https://mlsanity.com/paper/2406.17759/) (2024)
- [Memory3: Language Modeling with Explicit Memory](https://mlsanity.com/paper/2407.01178/) (2024)
- [Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization](https://mlsanity.com/paper/2410.12949/) (2024)
- [How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training](https://mlsanity.com/paper/2502.11196/) (2025)
- [Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline](https://mlsanity.com/paper/2505.20546/) (2025)
- [Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal](https://mlsanity.com/paper/2509.09708/) (2025)
- [Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information](https://mlsanity.com/paper/2502.14258/) (2025)
- [Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer](https://mlsanity.com/paper/2506.01115/) (2025)
- [Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience](https://mlsanity.com/paper/2406.01352/) (2024)
- [Levels of Analysis for Large Language Models](https://mlsanity.com/paper/2503.13401/) (2025)
- [Mechanistic Indicators of Understanding in Large Language Models](https://mlsanity.com/paper/2507.08017/) (2025)
- [Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval](https://mlsanity.com/paper/2512.03276/) (2025)
- [The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level](https://mlsanity.com/paper/2604.02178/) (2026)
- [Why Fine-Tuning Encourages Hallucinations and How to Fix It](https://mlsanity.com/paper/2604.15574/) (2026)
- [Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers](https://mlsanity.com/paper/2407.11542/) (2024)
- [ResiDual Transformer Alignment with Spectral Decomposition](https://mlsanity.com/paper/2411.00246/) (2024)
- [All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens](https://mlsanity.com/paper/2509.09650/) (2025)
- [Universal Redundancies in Time Series Foundation Models](https://mlsanity.com/paper/2602.01605/) (2026)
- [Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones](https://mlsanity.com/paper/2507.00322/) (2025)
- [Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models](https://mlsanity.com/paper/2510.06107/) (2025)
- [Mechanistic Understanding of Language Models in Syntactic Code Completion](https://mlsanity.com/paper/2502.18499/) (2025)
- [Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis](https://mlsanity.com/paper/2509.24164/) (2025)
- [Tracing Relational Knowledge Recall in Large Language Models](https://mlsanity.com/paper/2604.19934/) (2026)

## Abstract (arXiv)

How do transformer-based large language models (LLMs) store and retrieve knowledge? We focus on the most basic form of this task -- factual recall, where the model is tasked with explicitly surfacing stored facts in prompts of form `Fact: The Colosseum is in the country of'. We find that the mechanistic story behind factual recall is more complex than previously thought. It comprises several distinct, independent, and qualitatively different mechanisms that additively combine, constructively interfering on the correct attribute. We term this generic phenomena the additive motif: models compute through summing up multiple independent contributions. Each mechanism's contribution may be insufficient alone, but summing results in constructive interfere on the correct answer. In addition, we extend the method of direct logit attribution to attribute an attention head's output to individual source tokens. We use this technique to unpack what we call `mixed heads' -- which are themselves a pair of two separate additive updates from different source tokens.