mlsanityData to 7 Oct 2026

From Canonical Correlation Analysis to Self-supervised Graph Neural Networks

arXiv 2106.12484 ↗First posted 23 Jun 2021cs.LGcited by 138

Not on the map. Why?

Abstract (arXiv)

We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus on instance-level discrimination, we optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Compared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most importantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variant information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an understanding for the new objective which can be equivalently seen as an instantiation of the Information Bottleneck Principle under the self-supervised setting. Despite its simplicity, our method performs competitively on seven public graph datasets. The code is available at: https://github.com/hengruizhang98/CCA-SSG.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 23 Jun 2021.

Builds on

41 of 67 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
29 more
  1. 2016Variational Graph Auto-Encoderscited by 1,373
  2. 2019Contrastive Multiview Codingcited by 1,297
  3. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  4. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  5. 2018Deep Graph Infomaxcited by 1,101
  6. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  9. 2015Image-based Recommendations on Styles and Substitutescited by 702
  10. 2018MINE: Mutual Information Neural Estimationcited by 680
  11. 2020Contrastive Multi-View Representation Learning on Graphscited by 651
  12. 2020Deep Graph Contrastive Representation Learningcited by 535
  13. 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
  14. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  15. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 461
  16. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  17. 2019Diffusion Improves Graph Learningcited by 414
  18. 2020Whitening for Self-Supervised Representation Learningcited by 278
  19. 2020Contrastive Clusteringcited by 264
  20. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  21. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
  22. 2020Graph Representation Learning via Graphical Mutual Information Maximizationcited by 217
  23. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  24. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  25. 2021Self-supervised Graph-level Representation Learning with Local and Global Structurecited by 125
  26. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  27. 2020Contrastive Self-supervised Learning for Graph Classificationcited by 68
  28. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  29. 2016The deterministic information bottleneckcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 26; 2023: 31; 2024: 37; 2025: 32; 2026: 11.

138 papers.

Sort:

By area:

  1. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  2. 2024LLaGA: Large Language and Graph Assistantcited by 177
  3. 2023GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learnercited by 131
  4. 2022What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoderscited by 82
  5. 2021ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learningcited by 72
  6. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
  7. 2022Graph Data Augmentation for Graph Machine Learning: A Surveycited by 61
  8. 2022Heterogeneous Graph Masked Autoencoderscited by 47
  9. 2022Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrumcited by 43
  10. 2023Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphscited by 39
  11. 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 37
  12. 2022Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalizationcited by 35
38 more
  1. 2022Uncovering the Structural Fairness in Graph Contrastive Learningcited by 29
  2. 2022A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applicationscited by 23
  3. 2023SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Maskingcited by 23
  4. 2023Counterfactual Learning on Graphs: A Surveycited by 22
  5. 2024Rethinking Graph Masked Autoencoders through Alignment and Uniformitycited by 22
  6. 2022ImGCL: Revisiting Graph Contrastive Learning on Imbalanced Node Classificationcited by 21
  7. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  8. 2022Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representationscited by 21
  9. 2023Graph-level Representation Learning with Joint-Embedding Predictive Architecturescited by 21
  10. 2024GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learningcited by 21
  11. 2024SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samplescited by 20
  12. 2022GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysiscited by 19
  13. 2023HomoGCL: Rethinking Homophily in Graph Contrastive Learningcited by 18
  14. 2022Graph Soft-Contrastive Learning via Neighborhood Rankingcited by 16
  15. 2024GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Modelscited by 16
  16. 2024GraphAlign: Pretraining One Graph Neural Network on Multiple Graphs via Feature Alignmentcited by 15
  17. 2022Wiener Graph Deconvolutional Network Improves Graph Self-Supervised Learningcited by 13
  18. 2023Directional diffusion models for graph representation learningcited by 13
  19. 2023Rethinking Semi-Supervised Imbalanced Node Classification from Bias-Variance Decompositioncited by 13
  20. 2023Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspectivecited by 13
  21. 2024Reliable Node Similarity Matrix Guided Contrastive Graph Clusteringcited by 12
  22. 2022Relational Self-Supervised Learning on Graphscited by 11
  23. 2022Link Prediction with Non-Contrastive Learningcited by 11
  24. 2024Masked Graph Autoencoder with Non-discrete Bandwidthscited by 11
  25. 2024Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphscited by 11
  26. 2024CLDG: Contrastive Learning on Dynamic Graphscited by 11
  27. 2025Generative Graph Pattern Machinecited by 11
  28. 2022Decoupled Self-supervised Learning for Non-Homophilous Graphscited by 10
  29. 2022Localized Contrastive Learning on Graphscited by 10
  30. 2022MA-GCL: Model Augmentation Tricks for Graph Contrastive Learningcited by 10
  31. 2022Data Augmentation on Graphs: A Technical Surveycited by 10
  32. 2023RARE: Robust Masked Graph Autoencodercited by 10
  33. 2023Rethinking and Simplifying Bootstrapped Graph Latentscited by 10
  34. 2024Node Identifiers: Compact, Discrete Representations for Efficient Graph Learningcited by 10
  35. 2024Road Network Representation Learning with the Third Law of Geographycited by 10
  36. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  37. 2023Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learningcited by 9
  38. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9

50 most cited of 138 All 138 (JSON)