# Spectral Geometry of Attention: From Information Routing to Uncertainty

arXiv: https://arxiv.org/abs/2610.06012 · HTML: https://mlsanity.com/paper/2610.06012/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Why do LLMs attend to the first token?](https://mlsanity.com/paper/2504.02732/) (2025)
- [Transformers need glasses! Information over-squashing in language tasks](https://mlsanity.com/paper/2406.04267/) (2024)
- [LUQ: Long-text Uncertainty Quantification for LLMs](https://mlsanity.com/paper/2403.20279/) (2024)
- [Spectral Filters, Dark Signals, and Attention Sinks](https://mlsanity.com/paper/2402.09221/) (2024)
- [Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin](https://mlsanity.com/paper/2510.06477/) (2025)
- [Hallucination Detection in LLMs Using Spectral Features of Attention Maps](https://mlsanity.com/paper/2502.17598/) (2025)
- [What are you sinking? A geometric approach on attention sink](https://mlsanity.com/paper/2508.02546/) (2025)
- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [Hallucination Detection in LLMs with Topological Divergence on Attention Graphs](https://mlsanity.com/paper/2504.10063/) (2025)
- [Perplexity Cannot Always Tell Right from Wrong](https://mlsanity.com/paper/2601.22950/) (2026)
- [Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Method](https://mlsanity.com/paper/2601.20026/) (2026)
- [Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models](https://mlsanity.com/paper/2505.10606/) (2025)
- [A Graph Signal Processing Framework for Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2510.19117/) (2025)
- [SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis](https://mlsanity.com/paper/2412.19055/) (2024)

## Abstract (arXiv)

In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of functions on the token sequence and derive a Token Difference Operator, whose spectral structure controls how token-space information is routed to the output. We show that standard Euclidean spectra are structurally biased by sinks, conflating mass concentration with genuine routing capacity. By recasting token space in the intrinsic probability geometry induced by attention, the token difference spectrum disentangles sink effects from routing capacity and provides a spectral description of the dimensionality of the head output. This yields a unified framework for analyzing attention maps, explaining sinks, routing collapse, and output dimensionality within a single operator-theoretic framework. In practice, by grounding attention heuristics in spectral geometry, we develop a novel attention-based uncertainty estimator that complements probability-based scores, with the largest gains on long-context inputs.