Generalized Probabilistic Attention Mechanism in Transformers
Not on the map. Why?
Abstract (arXiv)
The Transformer architecture has become widely adopted due to its demonstrated success, attributed to the attention mechanism at its core. Despite these successes, the attention mechanism of Transformers is associated with two well-known issues: rank-collapse and gradient vanishing. In this paper, we present a theoretical analysis that it is inherently difficult to address both issues simultaneously in the conventional attention mechanism. To handle these issues, we introduce a novel class of attention mechanism, referred to as generalized probabilistic attention mechanism (GPAM), and its dual-attention implementation within the Transformer architecture. Unlike conventional attention mechanisms, GPAM allows for negative attention scores while preserving a fixed total sum. We provide theoretical evidence that the proposed dual-attention GPAM (daGPAM) effectively mitigates both the rank-collapse and gradient vanishing issues which are difficult to resolve simultaneously with the conventional attention mechanisms. Furthermore, we empirically validate this theoretical evidence, demonstrating the superiority of daGPAM compared to other alternative attention mechanisms that were proposed to address the same issues. Additionally, we demonstrate the practical benefits of GPAM in natural language processing tasks, such as language modeling and neural machine translation.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 21 Oct 2024.
Builds on
38 of 47 references.
References · 38
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
26 more
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
- 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
- 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
- 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 60
- 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
- 2022Rank Diminishing in Deep Neural Networkscited by 47
- 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
- 2021Vision Transformers with Patch Diversificationcited by 45
- 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
- 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
- 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
- 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
- 2020Normalized Attention Without Probability Cagecited by 20
- 2021Augmented Shortcuts for Vision Transformerscited by 19
- 2022Feature Overcorrelation in Deep Graph Neural Networks: A New Perspectivecited by 19
- 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
- 2022Addressing Token Uniformity in Transformers via Singular Value Transformationcited by 14
- 2021Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanismcited by 7
- 2024N-gram Prediction and Word Difference Representations for Language Modelingcited by 3
- 2023Why "classic" Transformers are shallow and how to make them go deepcited by 1
Built on by
Citing papers per year: 2025: 2.
2 papers.