Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink
Not on the map. Why?
Abstract (arXiv)
Spatio-temporal models analyze spatial structures and temporal dynamics, which makes them prone to information degeneration among space and time. Prior literature has demonstrated that over-squashing in causal attention or temporal convolutions creates a bias on the first tokens. To analyze whether such a bias is present in temporal attention mechanisms, we derive sensitivity bounds on the expected value of the Jacobian of a temporal attention layer. We theoretically show how off-diagonal attention scores depend on the sequence length, and that temporal attention matrices suffer a diagonal attention sink. We suggest regularization methods, and experimentally demonstrate their effectiveness.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 12 of its 14 references that are in the map. First posted on arXiv 11 Feb 2026.
Builds on
12 of 14 references.
References · 12
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
- 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
- 2024Transformers need glasses! Information over-squashing in language taskscited by 87
- 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
- 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
- 2023Improving Position Encoding of Transformers for Multivariate Time Series Classificationcited by 44
- 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 42
- 2024On Oversquashing in Graph Neural Networks Through the Lens of Dynamical Systemscited by 26
- 2023Feature Transportation Improves Graph Neural Networkscited by 15
- 2024Port-Hamiltonian Architectural Bias for Long-Range Propagation in Deep Graph Networkscited by 14
- 2025Over-squashing in Spatiotemporal Graph Neural Networkscited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.