mlsanityData to 14 Oct 2026

Centered Self-Attention Layers

arXiv 2306.01610 ↗First posted 2 Jun 2023cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures. We show that this application inevitably leads to oversmoothing, i.e., to similar representations at the deeper layers for different tokens in transformers and different nodes in graph neural networks. Based on our analysis, we present a correction term to the aggregating operator of these mechanisms. Empirically, this simple term eliminates much of the oversmoothing problem in visual transformers, obtaining performance in weakly supervised segmentation that surpasses elaborate baseline methods that introduce multiple auxiliary networks and training phrases. In graph neural networks, the correction term enables the training of very deep architectures more effectively than many recent solutions to the same problem.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 56 of its 62 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

56 of 62 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  7. 2017Graph Attention Networkscited by 8,923
  8. 2016Layer Normalizationcited by 5,669
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
38 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 922
  2. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2021ImageNet-21K Pretraining for the Massescited by 613
  5. 2019Multi-scale Attributed Node Embeddingcited by 548
  6. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  7. 2021GRAND: Graph Neural Diffusioncited by 258
  8. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 249
  9. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 247
  10. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  11. 2020Understanding the Difficulty of Training Transformerscited by 210
  12. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  13. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  14. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  15. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  16. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  17. 2021Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentationcited by 127
  18. 2019Pixel-Adaptive Convolutional Neural Networkscited by 126
  19. 2018On Regularized Losses for Weakly-supervised CNN Segmentationcited by 121
  20. 2022Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformerscited by 118
  21. 2022Not too little, not too much: a theoretical analysis of graph (over)smoothingcited by 106
  22. 2019Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approachcited by 88
  23. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85
  24. 2019Attention Guided Graph Convolutional Networks for Relation Extractioncited by 82
  25. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  26. 2021Reducing Information Bottleneck for Weakly Supervised Semantic Segmentationcited by 65
  27. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 65
  28. 2022Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentationcited by 64
  29. 2019Self-Supervised Difference Detection for Weakly-Supervised Semantic Segmentationcited by 58
  30. 2022Weakly Supervised Semantic Segmentation using Out-of-Distribution Datacited by 58
  31. 2022L2G: A Simple Local-to-Global Knowledge Transfer Framework for Weakly Supervised Semantic Segmentationcited by 57
  32. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  33. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  34. 2022A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networkscited by 49
  35. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 39
  36. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  37. 2022Expansion and Shrinkage of Localization for Weakly-Supervised Semantic Segmentationcited by 37
  38. 2023ResiDual: Transformer with Dual Residual Connectionscited by 31

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 2; 2026: 1.

7 papers.

  1. 2024The Hidden Attention of Mamba Modelscited by 91
  2. 2023Simplifying Transformer Blockscited by 37
  3. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  4. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  5. 2025Latent Mamba Operator for Partial Differential Equationscited by 10
  6. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottleneckscited by 3
  7. 2025Understanding Differential Transformer Unchains Pretrained Self-Attentionscited by 0