mlsanityData to 7 Oct 2026

Large-scale graph representation learning with very deep GNNs and self-supervision

arXiv 2107.09422 ↗First posted 20 Jul 2021cs.LG cs.AI cs.SIcited by 18

Not on the map. Why?

Abstract (arXiv)

Effectively and efficiently deploying graph neural networks (GNNs) at scale remains one of the most challenging aspects of graph representation learning. Many powerful solutions have only ever been validated on comparatively small datasets, often with counter-intuitive outcomes -- a barrier which has been broken by the Open Graph Benchmark Large-Scale Challenge (OGB-LSC). We entered the OGB-LSC with two large-scale GNNs: a deep transductive node classifier powered by bootstrapping, and a very deep (up to 50-layer) inductive graph regressor regularised by denoising objectives. Our models achieved an award-level (top-3) performance on both the MAG240M and PCQM4M benchmarks. In doing so, we demonstrate evidence of scalable self-supervised graph representation learning, and utility of very deep GNNs -- both very important open issues. Our code is publicly available at: https://github.com/deepmind/deepmind-research/tree/master/ogb_lsc.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 20 Jul 2021.

Builds on

43 of 66 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2017Graph Attention Networkscited by 8,854
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2017Inductive Representation Learning on Large Graphscited by 6,284
  9. 2016Layer Normalizationcited by 5,579
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  12. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
31 more
  1. 2019Simplifying Graph Convolutional Networkscited by 1,470
  2. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  3. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  5. 2018Deep Graph Infomaxcited by 1,101
  6. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  7. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  8. 2021E(n) Equivariant Graph Neural Networkscited by 951
  9. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  10. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  11. 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 718
  12. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  13. 2020Benchmarking Graph Neural Networkscited by 672
  14. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  15. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  16. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  17. 2020Deep Graph Contrastive Representation Learningcited by 535
  18. 2018Spherical CNNscited by 452
  19. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  20. 2020SIGN: Scalable Inception Graph Neural Networkscited by 283
  21. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 199
  22. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  23. 2020Scaling Graph Neural Networks with Approximate PageRankcited by 174
  24. 2019Neural Execution of Graph Algorithmscited by 118
  25. 2020Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphscited by 90
  26. 2020Chip Placement with Deep Reinforcement Learningcited by 83
  27. 2019GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulationcited by 75
  28. 2018Relational inductive bias for physical construction in humans and machinescited by 41
  29. 2021Bag of Tricks for Node Classification with Graph Neural Networkscited by 40
  30. 2020Scalable Graph Neural Networks for Heterogeneous Graphscited by 31
  31. 2018Graph Partition Neural Networks for Semi-Supervised Classificationcited by 17

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 5; 2023: 4; 2024: 3; 2025: 3.

18 papers.

Sort:

  1. 2022Ordered Subgraph Aggregation Networkscited by 57
  2. 2025PyG 2.0: Scalable Learning on Real World Graphscited by 49
  3. 2022Unified 2D and 3D Pre-Training of Molecular Representationscited by 45
  4. 2021On Representation Knowledge Distillation for Graph Neural Networkscited by 38
  5. 2024Fully-inductive Node Classification on Arbitrary Graphscited by 36
  6. 2021Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphscited by 31
  7. 2022Direct Molecular Conformation Generationcited by 30
  8. 2022Affinity-Aware Graph Networkscited by 17
  9. 2024A review of graph neural network applications in mechanics-related domainscited by 14
  10. 2023Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awarenesscited by 11
  11. 2021Graph Neural Network Training with Data Tieringcited by 8
  12. 2025AMR-Transformer: Enabling Efficient Long-range Interaction for Complex Neural Fluid Simulationcited by 7
6 more
  1. 2024Training-free Graph Neural Networks and the Power of Labels as Featurescited by 5
  2. 2022Extreme Acceleration of Graph Neural Network-based Prediction Models for Quantum Chemistrycited by 3
  3. 2025ASTRA: A Scene-aware TRAnsformer-based model for trajectory predictioncited by 2
  4. 2023Evolving Computation Graphscited by 0
  5. 2023Anticipating Technical Expertise and Capability Evolution in Research Communities using Dynamic Graph Transformerscited by 0
  6. 2023Graph Neural Networks-based Hybrid Framework For Predicting Particle Crushing Strengthcited by 0