# Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information

arXiv: https://arxiv.org/abs/2502.14258 · HTML: https://mlsanity.com/paper/2502.14258/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 39)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)

## Built on by (most cited first, 15 of 15)

- [TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions](https://mlsanity.com/paper/2502.21024/) (2025)
- [Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering](https://mlsanity.com/paper/2505.12826/) (2025)
- [Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training](https://mlsanity.com/paper/2509.25758/) (2025)
- [Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research](https://mlsanity.com/paper/2509.22831/) (2025)
- [Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity](https://mlsanity.com/paper/2511.21974/) (2025)
- [Time Travel Engine: A Shared Latent Chronological Manifold Enables Historical Navigation in Large Language Models](https://mlsanity.com/paper/2601.06437/) (2026)
- [ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack](https://mlsanity.com/paper/2509.25843/) (2025)
- [The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations](https://mlsanity.com/paper/2605.09195/) (2026)
- [PRISM Edit: One Vector for All Temporal Answers](https://mlsanity.com/paper/2607.11327/) (2026)
- [Do Language Models Consistently Encode the Current Year?](https://mlsanity.com/paper/2608.15507/) (2026)
- [Temporal Dependencies in In-Context Learning: The Role of Induction Heads](https://mlsanity.com/paper/2604.01094/) (2026)
- [Revealing Algorithmic Deductive Circuits for Logical Reasoning](https://mlsanity.com/paper/2605.27824/) (2026)
- [Can LLMs Be Constrained to the Past? Improving Knowledge Cutoff through Recall-Based Prompting](https://mlsanity.com/paper/2606.05804/) (2026)
- [Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall](https://mlsanity.com/paper/2609.33183/) (2026)
- [Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency](https://mlsanity.com/paper/2610.13835/) (2026)

## Abstract (arXiv)

While the ability of language models to elicit facts has been widely investigated, how they handle temporally changing facts remains underexplored. We discover Temporal Heads, specific attention heads that primarily handle temporal knowledge, through circuit analysis. We confirm that these heads are present across multiple models, though their specific locations may vary, and their responses differ depending on the type of knowledge and its corresponding years. Disabling these heads degrades the model's ability to recall time-specific knowledge while maintaining its general capabilities without compromising time-invariant and question-answering performances. Moreover, the heads are activated not only numeric conditions ("In 2004") but also textual aliases ("In the year ..."), indicating that they encode a temporal dimension beyond simple numerical representation. Furthermore, we expand the potential of our findings by demonstrating how temporal knowledge can be edited by adjusting the values of these heads.