mlsanityData to 14 Oct 2026

The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations

arXiv 2605.09195 ↗First posted 9 May 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models confidently produce outdated answers, and no existing method can detect them. We show this is not an engineering failure but a structural one: temporal drift, whether a stored fact has changed since training, is encoded as a direction in the residual stream geometrically orthogonal to both correctness and uncertainty. Any method operating on correctness or uncertainty signals is therefore blind to drift by construction. We verify this across six instruction-tuned models. A linear probe trained directly on drift labels achieves AUROC 0.83--0.95; methods based on token entropy, semantic entropy, CCS, and SAPLMA all remain near chance (0.49--0.57). Five tests confirm the geometric orthogonality: weight cosines (|cos| ≤ 0.14), score correlations (|r| ≤ 0.20), bidirectional null-space projection (|Delta| ≤ 0.008), iterative null-space projection with k=10, and difference-of-means dissociation. Mechanistically, the MLP retrieval circuit produces identical dynamics for stale recall and confabulation (r > 0.81, six models), explaining why output confidence cannot separate them. A cross-cutoff experiment holds inputs constant and varies only the model: the probe fires on the model whose training predates the fact's transition and stays silent otherwise (P(A>B) = 0.975--0.998, twelve model pairs), confirming it reads model-internal knowledge state rather than input properties. Our code and datasets will be publicly released.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 38 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

36 of 38 references.

References · 36

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2022Mass-Editing Memory in a Transformercited by 1,001
  9. 2023Steering Language Models With Activation Engineeringcited by 922
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
24 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  5. 2023Active Retrieval Augmented Generationcited by 516
  6. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  7. 2022Memory-Based Model Editing at Scalecited by 447
  8. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  9. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  10. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  11. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  12. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  13. 2024How to use and interpret activation patchingcited by 214
  14. 2021A Dataset for Answering Time-Sensitive Questionscited by 154
  15. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  16. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
  17. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 51
  18. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 38
  19. 2025Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Informationcited by 15
  20. 2023An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4Lcited by 11
  21. 2026Closing the Confidence-Faithfulness Gap in Large Language Modelscited by 8
  22. 2026Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoderscited by 6
  23. 2026The Confidence Manifold: Geometric Structure of Correctness Representations in Language Modelscited by 4
  24. 2025Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanismscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can an LLM Induce a Graph? Investigating Memory Drift and Context Length
  2. 2026Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability
  3. 2026Beyond Fixed Directions: Adaptive Representation Analysis of Reasoning and Memorization in LLMs
  4. 2025Forgetting as a Feature: Cognitive Alignment of Large Language Models
  5. 2026What Should World Models Forget? Stratified Retention for Continual Adaptation
  6. 2026Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
  7. 2026MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
  8. 2026Representation Drift in Continual Learning: LoRA versus N-gram Memory