On Understanding and Mitigating the Dimensional Collapse of Graph Contrastive Learning: a Non-Maximum Removal Approach
Not on the map. Why?
Abstract (arXiv)
Graph Contrastive Learning (GCL) has shown promising performance in graph representation learning (GRL) without the supervision of manual annotations. GCL can generate graph-level embeddings by maximizing the Mutual Information (MI) between different augmented views of the same graph (positive pairs). However, the GCL is limited by dimensional collapse, i.e., embedding vectors only occupy a low-dimensional subspace. In this paper, we show that the smoothing effect of the graph pooling and the implicit regularization of the graph convolution are two causes of the dimensional collapse in GCL. To mitigate the above issue, we propose a non-maximum removal graph contrastive learning approach (nmrGCL), which removes "prominent'' dimensions (i.e., contribute most in similarity measurement) for positive pair in the pre-text task. Comprehensive experiments on various benchmark datasets are conducted to demonstrate the effectiveness of nmrGCL, and the results show that our model outperforms the state-of-the-art methods. Source code will be made publicly available.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 24 Mar 2022.
Builds on
39 of 51 references.
References · 39
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Graph Attention Networkscited by 8,923
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2017Inductive Representation Learning on Large Graphscited by 6,320
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2018How Powerful are Graph Neural Networks?cited by 4,393
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
- 2020Graph Contrastive Learning with Augmentationscited by 1,208
- 2017Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecastingcited by 1,136
27 more
- 2018Deep Graph Infomaxcited by 1,106
- 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
- 2018Pitfalls of Graph Neural Network Evaluationcited by 1,007
- 2019Strategies for Pre-training Graph Neural Networkscited by 971
- 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 728
- 2019Graph Neural Networks for Social Recommendationcited by 714
- 2020Contrastive Multi-View Representation Learning on Graphscited by 652
- 2020Deep Graph Contrastive Representation Learningcited by 539
- 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
- 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 463
- 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 443
- 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
- 2017Implicit Regularization in Matrix Factorizationcited by 322
- 2021Graph Contrastive Learning Automatedcited by 287
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
- 2017graph2vec: Learning Distributed Representations of Graphscited by 230
- 2020Graph Representation Learning via Graphical Mutual Information Maximizationcited by 217
- 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
- 2020Directional convergence and alignment in deep learningcited by 178
- 2018Adversarial Complementary Learning for Weakly Supervised Object Localizationcited by 166
- 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
- 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 141
- 2021An Empirical Study of Graph Contrastive Learningcited by 113
- 2016Squared Earth Mover's Distance-based Loss for Training Deep Neural Networkscited by 45
- 2016Wasserstein Discriminant Analysiscited by 25
Built on by
Citing papers per year: 2025: 1.
1 paper.