mlsanityData to 7 Oct 2026

Prescriptive SVD-Inspired Attention via Spectral Energy Retention

arXiv 2609.24370 ↗First posted 21 Sep 2026cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-attention is central to modern Transformer architectures, but its dense dot-product formulation makes it difficult to identify which internal directions are structurally important and which can be modified without disrupting the model. SVD-Inspired Attention (SVDA) addresses part of this problem by introducing a learned diagonal spectrum into the query-key score interaction, making latent attention directions explicitly inspectable through indicators such as spectral entropy, effective rank, sparsity, alignment, selectivity, and perturbation response. This paper examines the transition from diagnostic interpretation to operational intervention. A diagnosis--intervention--verification framework is proposed, and one intervention is evaluated: spectral energy retention in the attention-score pathway. Across FashionMNIST, CIFAR-10, CIFAR-100, and Food-101, the ρ=0.90 prescription removes 24.5--53.7% of score directions, reduces parameters by 2.6--4.3%, and reduces estimated MACs by 2.8--5.4%. The paired mean accuracy change of the dimension-reduced model ranges from -0.03 to +0.05 percentage points over three seeds. These results support SVDA as an intrinsically interpretable attention mechanism whose learned spectrum exposes an operational coordinate system for deterministic and verifiable modification of attention-score formation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

31 of 39 references.

References · 31

  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  11. 2019Attention is not not Explanationcited by 629
  12. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
19 more
  1. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  2. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  3. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  4. 2019Adaptively Sparse Transformerscited by 154
  5. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  6. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  7. 2023Specformer: Spectral Graph Neural Networks Meet Transformerscited by 102
  8. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  9. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  10. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  11. 2024Interpretable Image Classification with Adaptive Prototype-based Vision Transformerscited by 25
  12. 2020Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Trainingcited by 12
  13. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12
  14. 2024Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustnesscited by 10
  15. 2025MDP: Multidimensional Vision Model Pruning with Latency Constraintcited by 4
  16. 2025HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformercited by 2
  17. 2022Verifying Attention Robustness of Deep Neural Networks against Semantic Perturbationscited by 1
  18. 2026Interpretable Vision Transformers in Image Classification via SVDAcited by 1
  19. 2026Interpretable Vision Transformers in Monocular Depth Estimation via SVDAcited by 1

Built on by

Not cited yet.