mlsanityData to 7 Oct 2026

Spectral Conditioning of Attention Improves Transformer Performance

arXiv 2603.07162 ↗First posted 7 Mar 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the attention mechanism. Leveraging this insight, we introduce a method that systematically alters the spectral properties of each attention layer to reduce the Jacobian's condition number, thereby improving the overall conditioning of the attention layers within a transformer network. We empirically show that this improved Jacobian conditioning translates to enhanced performance in practice. Our approach is simple, broadly applicable, and can be easily integrated as a drop-in replacement for a wide range of existing attention mechanisms. We validate its effectiveness across diverse transformer architectures and tasks, demonstrating consistent improvements in performance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

30 of 42 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
18 more
  1. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  2. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  3. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  4. 2022DaViT: Dual Attention Vision Transformerscited by 174
  5. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  6. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  7. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  8. 2020Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Datacited by 58
  9. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  10. 2020A Deep Conditioning Treatment of Neural Networkscited by 15
  11. 2025Always Skip Attentioncited by 12
  12. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Productcited by 10
  13. 2024Weight Conditioning for Smooth Optimization of Neural Networkscited by 9
  14. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  15. 2025Cutting the Skip: Training Residual-Free Transformerscited by 7
  16. 2025Structured Initialization for Vision Transformerscited by 6
  17. 2025Leaner Transformers: More Heads, Less Depthcited by 5
  18. 2025Enhancing Transformers Through Conditioned Embedded Tokenscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topologycited by 5
  2. 2026Conditioned Initialization for Attentioncited by 0
  3. 2026NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformerscited by 0