mlsanityData to 7 Oct 2026

Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning

arXiv 2311.02687 ↗First posted 5 Nov 2023cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

With the prosperity of contrastive learning for visual representation learning (VCL), it is also adapted to the graph domain and yields promising performance. However, through a systematic study of various graph contrastive learning (GCL) methods, we observe that some common phenomena among existing GCL methods that are quite different from the original VCL methods, including 1) positive samples are not a must for GCL; 2) negative samples are not necessary for graph classification, neither for node classification when adopting specific normalization modules; 3) data augmentations have much less influence on GCL, as simple domain-agnostic augmentations (e.g., Gaussian noise) can also attain fairly good performance. By uncovering how the implicit inductive bias of GNNs works in contrastive learning, we theoretically provide insights into the above intriguing properties of GCL. Rather than directly porting existing VCL methods to GCL, we advocate for more attention toward the unique architecture of graph learning and consider its implicit influence when designing GCL methods. Code is available at https: //github.com/PKU-ML/ArchitectureMattersGCL.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 62 of its 80 references that are in the map. First posted on arXiv 5 Nov 2023.

Builds on

62 of 80 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2018How Powerful are Graph Neural Networks?cited by 4,355
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
38 more
  1. 2019Simplifying Graph Convolutional Networkscited by 1,470
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  4. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  5. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  6. 2018Deep Graph Infomaxcited by 1,101
  7. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  10. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  11. 2020Contrastive Multi-View Representation Learning on Graphscited by 651
  12. 2019Multi-scale Attributed Node Embeddingcited by 543
  13. 2020Jukebox: A Generative Model for Musiccited by 535
  14. 2020Deep Graph Contrastive Representation Learningcited by 535
  15. 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
  16. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  17. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  18. 2021Graph Contrastive Learning Automatedcited by 287
  19. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  20. 2021Large-Scale Representation Learning on Graphs via Bootstrappingcited by 184
  21. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  22. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  23. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  24. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  25. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  26. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  27. 2021Graph Barlow Twins: A self-supervised representation learning framework for graphscited by 85
  28. 2021AutoGCL: Automated Graph Contrastive Learning via Learnable View Generatorscited by 73
  29. 2021ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learningcited by 72
  30. 2022Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discriminationcited by 67
  31. 2022Spectral Feature Augmentation for Graph Contrastive Learning and Beyondcited by 49
  32. 2022Message passing all the way upcited by 47
  33. 2022Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrumcited by 43
  34. 2022Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPscited by 42
  35. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  36. 2022Spectral Augmentation for Self-Supervised Learning on Graphscited by 38
  37. 2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
  38. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  2. 2024Rethinking Spectral Augmentation for Contrast-based Graph Self-Supervised Learningcited by 1
  3. 2025DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalizationcited by 1
  4. 2025Node Embeddings via Neighbor Embeddingscited by 1
  5. 2025HGCL: Hierarchical Graph Contrastive Learning for User-Item Recommendationcited by 1
  6. 2025Graph Contrastive Learning versus Untrained Baselines: The Role of Dataset Sizecited by 0