mlsanityData to 14 Oct 2026

ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond

arXiv 2303.06562 ↗First posted 12 Mar 2023cs.LG cs.CV stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance worsens as the number of layers increases. Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective of dimensional collapse in which representations lie in a narrow cone. Accordingly, inspired by the effectiveness of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer called ContraNorm. Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and a slighter dimensional collapse. On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under certain conditions. Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead. Experiments on various real-world datasets demonstrate the effectiveness of our proposed ContraNorm. Our implementation is available at https://github.com/PKU-ML/ContraNorm.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

55 of 69 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2016Layer Normalizationcited by 5,669
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  10. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  11. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  13. 2020Simple and Deep Graph Convolutional Networkscited by 860
  14. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  15. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  16. 2019Multi-scale Attributed Node Embeddingcited by 548
  17. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  18. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  19. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  20. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  21. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  22. 2020Whitening for Self-Supervised Representation Learningcited by 279
  23. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  24. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  25. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  26. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  27. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  28. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  29. 2021Interpreting and Unifying Graph Neural Networks with An Optimization Frameworkcited by 131
  30. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  31. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  32. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  33. 2018Decorrelated Batch Normalizationcited by 83
  34. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  35. 2021Graph Neural Networks Inspired by Classical Iterative Algorithmscited by 60
  36. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  37. 2021Vision Transformers with Patch Diversificationcited by 45
  38. 2022Optimization-Induced Graph Implicit Nonlinear Diffusioncited by 28

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 12; 2025: 16; 2026: 1.

39 papers.

Sort:

By area:

  1. 2024CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patchingcited by 94
  2. 2024Fredformer: Frequency Debiased Transformer for Time Series Forecastingcited by 45
  3. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 34
  4. 2024FairGT: A Fairness-aware Graph Transformercited by 29
  5. 2024Measure-to-measure interpolation using Transformerscited by 29
  6. 2023An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attentioncited by 28
  7. 2024Do Generated Data Always Help Contrastive Learning?cited by 25
  8. 2023Towards Deep Attention in Graph Neural Networks: Problems and Remediescited by 24
  9. 2024Rethinking Graph Masked Autoencoders through Alignment and Uniformitycited by 22
  10. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  11. 2024Non-negative Contrastive Learningcited by 19
  12. 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
27 more
  1. 2023Identifiable Contrastive Learning with Automatic Feature Importance Discoverycited by 14
  2. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  3. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  4. 2024GTC: GNN-Transformer Co-contrastive Learning for Self-supervised Heterogeneous Graph Representationcited by 9
  5. 2023Centered Self-Attention Layerscited by 7
  6. 2023Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learningcited by 7
  7. 2024Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognitioncited by 7
  8. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  9. 2025STAR: Boosting Time Series Foundation Models for Anomaly Detection through State-aware Adaptercited by 7
  10. 2025Deeper with Riemannian Geometry: Overcoming Oversmoothing and Oversquashing for Graph Foundation Modelscited by 7
  11. 2025Are We Measuring Oversmoothing in Graph Neural Networks Correctly?cited by 6
  12. 2025MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scorescited by 4
  13. 2025Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTexcited by 3
  14. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  15. 2024Generalized Probabilistic Attention Mechanism in Transformerscited by 2
  16. 2025A Signed Graph Approach to Understanding and Mitigating Oversmoothing in GNNscited by 2
  17. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  18. 2025Fairness in Augmented Graph Learning: A Surveycited by 2
  19. 2023Polynomial-based Self-Attention for Table Representation learningcited by 1
  20. 2023Degree-based stratification of nodes in Graph Neural Networkscited by 1
  21. 2025On the Role of Hidden States of Modern Hopfield Network in Transformercited by 1
  22. 2023Oversmoothing: A Nightmare for Graph Contrastive Learning?cited by 0
  23. 2025Channel-Attentive Graph Neural Networkscited by 0
  24. 2025Simplifying Graph Convolutional Networks with Redundancy-Free Neighborscited by 0
  25. 2025Identifying and Understanding Cross-Class Features in Adversarial Trainingcited by 0
  26. 2025Towards a deeper GCN: Alleviate over-smoothing with iterative training and fine-tuningcited by 0
  27. 2026Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulationcited by 0