mlsanityData to 7 Oct 2026

Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods

arXiv 2111.04840 ↗First posted 8 Nov 2021cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

Graph Neural Networks (GNNs) have achieved state-of-the-art performance in node classification, regression, and recommendation tasks. GNNs work well when rich and high-quality connections are available. However, their effectiveness is often jeopardized in many real-world graphs in which node degrees have power-law distributions. The extreme case of this situation, where a node may have no neighbors, is called Strict Cold Start (SCS). SCS forces the prediction to rely completely on the node's own features. We propose Cold Brew, a teacher-student distillation approach to address the SCS and noisy-neighbor challenges for GNNs. We also introduce feature contribution ratio (FCR), a metric to quantify the behavior of inductive GNNs to solve SCS. We experimentally show that FCR disentangles the contributions of different graph data components and helps select the best architecture for SCS generalization. We further demonstrate the superior performance of Cold Brew on several public benchmark and proprietary e-commerce datasets, where many nodes have either very few or noisy connections. Our source code is available at https://github.com/amazon-research/gnn-tail-generalization.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

45 of 57 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Graph Attention Networkscited by 8,854
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2017Inductive Representation Learning on Large Graphscited by 6,284
  7. 2018How Powerful are Graph Neural Networks?cited by 4,355
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2019Simplifying Graph Convolutional Networkscited by 1,470
  11. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  12. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
33 more
  1. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  2. 2020Simple and Deep Graph Convolutional Networkscited by 857
  3. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  4. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  5. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  6. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  7. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  8. 2020Towards Deeper Graph Neural Networkscited by 296
  9. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  10. 2019Revisiting Graph Neural Networks: All We Have is Low-Pass Filterscited by 260
  11. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  12. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 199
  13. 2017Graph Embedding Techniques, Applications, and Performance: A Surveycited by 188
  14. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  15. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  16. 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 113
  17. 2020Unifying Graph Convolutional Neural Networks and Label Propagationcited by 98
  18. 2020Non-Local Graph Neural Networkscited by 91
  19. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  20. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  21. 2020Tackling Over-Smoothing for General Graph Convolutional Networkscited by 66
  22. 2019Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networkscited by 62
  23. 2021Dissecting the Diffusion Process in Linear Graph Convolutional Networkscited by 58
  24. 2020Revisiting Over-smoothing in Deep GCNscited by 55
  25. 2019STAR-GCN: Stacked and Reconstructed Graph Convolutional Networks for Recommender Systemscited by 54
  26. 2020Understanding and Resolving Performance Degradation in Graph Convolutional Networkscited by 53
  27. 2021Zero-Shot Recommender Systemscited by 45
  28. 2020Pre-Training Graph Neural Networks for Cold-Start Users and Items Representationcited by 42
  29. 2020On Self-Distilling Graph Neural Networkcited by 35
  30. 2019From Zero-Shot Learning to Cold-Start Recommendationcited by 29
  31. 2020Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspectivecited by 17
  32. 2018Understanding Regularized Spectral Clustering via Graph Conductancecited by 14
  33. 2020Recommendation system using a deep learning and graph analysis approachcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 21; 2024: 13; 2025: 8; 2026: 4.

57 papers.

Sort:

By area:

  1. 2022A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinkingcited by 56
  2. 2022Linkless Link Prediction via Relational Distillationcited by 39
  3. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  4. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPscited by 34
  5. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphscited by 29
  6. 2023GraphPatcher: Mitigating Degree Bias for Graph Neural Networks via Test-time Augmentationcited by 29
  7. 2022Infinite Recommendation Networks: A Data-Centric Approachcited by 20
  8. 2023Modeling Dynamic Environments with Scene Graph Memorycited by 18
  9. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  10. 2024LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learningcited by 17
  11. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
  12. 2023Graph Collaborative Signals Denoising and Augmentation for Recommendationcited by 16
38 more
  1. 2023A Survey of Imbalanced Learning on Graphs: Problems, Techniques, and Future Directionscited by 15
  2. 2022NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphscited by 14
  3. 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillationcited by 14
  4. 2023Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendationcited by 13
  5. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLPcited by 12
  6. 2022Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Againcited by 11
  7. 2024Optimizing Long-tailed Link Prediction in Graph Neural Networks through Structure Representation Enhancementcited by 9
  8. 2023Collaborative Graph Neural Networks for Attributed Network Embeddingcited by 8
  9. 2024Training MLPs on Graphs without Supervisioncited by 8
  10. 2024GraphStorm: all-in-one graph machine learning framework for industry applicationscited by 7
  11. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6
  12. 2023SAILOR: Structural Augmentation Based Tail Node Representation Learningcited by 6
  13. 2024A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learningcited by 5
  14. 2023You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Predictioncited by 4
  15. 2024Node Duplication Improves Cold-start Link Predictioncited by 4
  16. 2024TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillationcited by 4
  17. 2024CNN2GNN: How to Bridge CNN with GNNcited by 3
  18. 2025Incomplete Graph Learning: A Comprehensive Surveycited by 3
  19. 2025Aggregation Buffer: Revisiting DropEdge with a New Parameter Blockcited by 3
  20. 2023OrthoReg: Improving Graph-regularized MLPs via Orthogonality Regularizationcited by 2
  21. 2023Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communicationcited by 2
  22. 2024Graph Knowledge Distillation to Mixture of Expertscited by 2
  23. 2025HI-PMK: A Data-Dependent Kernel for Incomplete Heterogeneous Data Representationcited by 2
  24. 2023Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillationcited by 1
  25. 2023Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPscited by 1
  26. 2024HyperAggregation: Aggregating over Graph Edges with Hypernetworkscited by 1
  27. 2024SPARC: Spectral Architectures Tackling the Cold-Start Problem in Graph Learningcited by 1
  28. 2024Scale-aware Message Passing For Graph Node Classificationcited by 1
  29. 2025Norm Augmented Graph AutoEncoders for Link Predictioncited by 1
  30. 2025Effects of Dropout on Performance in Long-range Graph Learning Taskscited by 1
  31. 2025Tuning-Free LLM Can Build A Strong Recommender Under Sparse Connectivity And Knowledge Gap Via Extracting Intentcited by 1
  32. 2026Pruned Traffic Trees: Native Semantic Compression with a Protocol-Structured Model Family for Encrypted Traffic Classificationcited by 1
  33. 2022TuneUp: A Simple Improved Training Strategy for Graph Neural Networkscited by 0
  34. 2022Symbolic Distillation for Learned TCP Congestion Controlcited by 0
  35. 2022GLINKX: A Scalable Unified Framework For Homophilous and Heterophilous Graphscited by 0
  36. 2023Structural Imbalance Aware Graph Augmentation Learningcited by 0
  37. 2023Graph Entropy Minimization for Semi-supervised Node Classificationcited by 0
  38. 2023Unleashing the potential of GNNs via Bi-directional Knowledge Transfercited by 0

50 most cited of 57 All 57 (JSON)