mlsanityData to 7 Oct 2026

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

arXiv 2605.04893 ↗First posted 6 May 2026cs.LG cs.CL stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Every attention head defines a degree-normalized transport operator, and a growing family of diagnostics reads model behavior (hallucination among them) from its spectrum. We ask what such diagnostics can and cannot infer. The operator splits orthogonally into a symmetric part governing transport capacity and an antisymmetric part encoding orientation. We prove an identifiability limit: every transpose-invariant spectral diagnostic is orientation-blind (unable to distinguish an operator from its transpose, hence blind to the orientation of information flow), with a transpose-stability bound limiting any Lipschitz diagnostic's transpose sensitivity by the asymmetry coefficient G. This bounds what spectral diagnostics of the attention operator can resolve (e.g. LapEigvals and the attention-spectral branch of LLM-Check). On the surviving axis, a closed-form bipartite-Cheeger landscape shows uniform causal attention obeys an n-independent temporal-cut floor φ ≥ 1/5 while window attention pierces it as O(w/n); the floor is an idealized benchmark, not an empirical attractor, and the fraction of real heads falling below it is itself an empirically stable architectural descriptor. The two-axis diagnostic (φ for capacity, G for asymmetry magnitude) yields a falsifiable polarity prediction, borne out in sign under length-controlled, forced-scoring evaluation across decoder-only, encoder-only, and encoder--decoder models (capacity-axis signal 0.62--0.84 LC-AUROC): polarity reverses between HaluEval and MedHallu, directionally as predicted though asymmetric in strength, with decision polarity calibrated per regime.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

30 of 57 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  11. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  12. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
18 more
  1. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  2. 2024Long-form factuality in large language modelscited by 146
  3. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  4. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  5. 2023The emergence of clusters in self-attention dynamicscited by 133
  6. 2025Hallucination Detection in LLMs Using Spectral Features of Attention Mapscited by 42
  7. 2025MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Modelscited by 33
  8. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  9. 2019Hermitian matrices for clustering directed graphs: insights and applicationscited by 27
  10. 2025The Illusion of Progress: Re-evaluating Hallucination Detection in LLMscited by 22
  11. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  12. 2025Hallucination Detection in LLMs with Topological Divergence on Attention Graphscited by 18
  13. 2022Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translationcited by 13
  14. 2025Attention (as Discrete-Time Markov) Chainscited by 11
  15. 2025EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMscited by 10
  16. 2025The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer trainingcited by 9
  17. 2013Improved Cheeger's Inequality: Analysis of Spectral Partitioning Algorithms through Higher Order Spectral Gapcited by 8
  18. 2022Cheeger Inequalities for Directed Graphs and Hypergraphs Using Reweighted Eigenvaluescited by 3

Built on by

Not cited yet.