Attention Is Not Retention: The Orthogonality Constraint in Infinite-Context Architectures
Not on the map. Why?
Abstract (arXiv)
Biological memory solves a problem that eludes current AI: storing specific episodic facts without corrupting general semantic knowledge. Complementary Learning Systems theory explains this through two subsystems - a fast hippocampal system using sparse, pattern-separated representations for episodes, and a slow neocortical system using distributed representations for statistical regularities. Current AI systems lack this separation, attempting to serve both functions through neural weights alone. We identify the Orthogonality Constraint: reliable memory requires orthogonal keys, but semantic embeddings cannot be orthogonal because training clusters similar concepts together. The result is Semantic Interference (connecting to what cognitive psychologists have long observed in human memory), where neural systems writing facts into shared continuous parameters collapse to near-random accuracy within tens of semantically related facts. Through semantic density (rho), the mean pairwise cosine similarity, we show collapse occurs at N=5 facts (rho > 0.6) or N ~ 20-75 (moderate rho). We validate across modalities: 16,309 Wikipedia facts, scientific measurements (rho = 0.96, 0.02% accuracy at N=10,000), and image embeddings (rho = 0.82, 0.05% at N=2,000). This failure is geometric - no increase in model capacity can overcome interference when keys share semantic overlap. We propose Knowledge Objects (KOs): structured facts with hash-based identity, controlled vocabularies, and explicit version chains. On Wikipedia facts, KO retrieval achieves 45.7% where Modern Hopfield Networks collapse to near-zero; hash-based retrieval maintains 100%. Production systems (Claude Memory, ChatGPT Memory) store unstructured text, causing schema drift (40-70% consistency) and version ambiguity. Knowledge Objects provide the discrete hippocampal component that enables reliable bicameral memory.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 9 of its 21 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
9 of 21 references.
References · 9
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,433
- 2014Neural Turing Machinescited by 639
- 2024Titans: Learning to Memorize at Test Timecited by 312
- 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 71
- 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
Built on by
Citing papers per year: 2026: 1.
1 paper.