mlsanityData to 7 Oct 2026

Revisiting Over-smoothing in BERT from the Perspective of Graph

arXiv 2202.08625 ↗First posted 17 Feb 2022cs.LGcited by 71

Not on the map. Why?

Abstract (arXiv)

Recently over-smoothing phenomenon of Transformer-based models is observed in both vision and language fields. However, no existing work has delved deeper to further investigate the main cause of this phenomenon. In this work, we make the attempt to analyze the over-smoothing problem from the perspective of graph, where such problem was first discovered and explored. Intuitively, the self-attention matrix can be seen as a normalized adjacent matrix of a corresponding graph. Based on the above connection, we provide some theoretical analysis and find that layer normalization plays a key role in the over-smoothing issue of Transformer-based models. Specifically, if the standard deviation of layer normalization is sufficiently large, the output of Transformer stacks will converge to a specific low-rank subspace and result in over-smoothing. To alleviate the over-smoothing problem, we consider hierarchical fusion strategies, which combine the representations from different layers adaptively to make the output more diverse. Extensive experiment results on various data sets illustrate the effect of our fusion method.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

36 of 50 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2016Layer Normalizationcited by 5,579
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
24 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  11. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  12. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  13. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  14. 2019Revealing the Dark Secrets of BERTcited by 300
  15. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  16. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  17. 2018Scaling Neural Machine Translationcited by 204
  18. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  19. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  20. 2020Tackling Over-Smoothing for General Graph Convolutional Networkscited by 66
  21. 2020RealFormer: Transformer Likes Residual Attentioncited by 62
  22. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  23. 2018Multi-granularity hierarchical attention fusion networks for reading comprehension and question answeringcited by 28
  24. 2019SANVis: Visual Analytics for Understanding Self-Attention Networkscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 18; 2025: 20; 2026: 9.

71 papers.

Sort:

By area:

  1. 2023Specformer: Spectral Graph Neural Networks Meet Transformerscited by 102
  2. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 64
  3. 2023Demystifying Oversmoothing in Attention-Based Graph Neural Networkscited by 47
  4. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  5. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  6. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  7. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 31
  8. 2022Transformers from an Optimization Perspectivecited by 27
  9. 2024Value Residual Learningcited by 25
  10. 2023Towards Deep Attention in Graph Neural Networks: Problems and Remediescited by 24
  11. 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
  12. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
38 more
  1. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  2. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  3. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  4. 2023Are More Layers Beneficial to Graph Transformers?cited by 14
  5. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
  6. 2023Decomposed Cross-modal Distillation for RGB-based Temporal Action Detectioncited by 12
  7. 2025A Survey of Graph Transformers: Architectures, Theories and Applicationscited by 12
  8. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  9. 2024PIDformer: Transformer Meets Control Theorycited by 10
  10. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  11. 2025Synchronization of mean-field models on the circlecited by 10
  12. 2022Modeling Fine-grained Information via Knowledge-aware Hierarchical Graph for Zero-shot Entity Retrievalcited by 9
  13. 2025Revisiting Kernel Attention with Correlated Gaussian Process Representationcited by 9
  14. 2023Centered Self-Attention Layerscited by 7
  15. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  16. 2024Elliptical Attentioncited by 7
  17. 2025Multistability of Self-Attention Dynamics in Transformerscited by 6
  18. 2024MultiMax: Sparse and Multi-Modal Attention Learningcited by 5
  19. 2024Multi-task Heterogeneous Graph Learning on Electronic Health Recordscited by 5
  20. 2022A Study on Transformer Configuration and Training Objectivecited by 4
  21. 2022AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuningcited by 4
  22. 2023Transformers are efficient hierarchical chemical graph learnerscited by 4
  23. 2024Tackling Ambiguity from Perspective of Uncertainty Inference and Affinity Diversification for Weakly Supervised Semantic Segmentationcited by 4
  24. 2024Lambda-Skip Connections: the architectural component that prevents Rank Collapsecited by 4
  25. 2025MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scorescited by 4
  26. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 4
  27. 2024Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modelingcited by 3
  28. 2024Zipfian Whiteningcited by 3
  29. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  30. 2025Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlationcited by 3
  31. 2025CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Passcited by 3
  32. 2025Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domaincited by 3
  33. 2026Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoningcited by 3
  34. 2023Masked Collaborative Contrast for Weakly Supervised Semantic Segmentationcited by 2
  35. 2024SIBO: A Simple Booster for Parameter-Efficient Fine-Tuningcited by 2
  36. 2024Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspectivecited by 2
  37. 2024Graph Neural Networks Need Cluster-Normalize-Activate Modulescited by 2
  38. 2025Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Modelscited by 2

50 most cited of 71 All 71 (JSON)