mlsanityData to 14 Oct 2026

Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads

arXiv 2105.11118 ↗First posted 24 May 2021cs.DC cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

A graph neural network (GNN) enables deep learning on structured graph data. There are two major GNN training obstacles: 1) it relies on high-end servers with many GPUs which are expensive to purchase and maintain, and 2) limited memory on GPUs cannot scale to today's billion-edge graphs. This paper presents Dorylus: a distributed system for training GNNs. Uniquely, Dorylus can take advantage of serverless computing to increase scalability at a low cost. The key insight guiding our design is computation separation. Computation separation makes it possible to construct a deep, bounded-asynchronous pipeline where graph and tensor parallel tasks can fully overlap, effectively hiding the network latency incurred by Lambdas. With the help of thousands of Lambda threads, Dorylus scales GNN training to billion-edge graphs. Currently, for large graphs, CPU servers offer the best performance-per-dollar over GPU servers. Just using Lambdas on top of CPU servers offers up to 2.75x more performance-per-dollar than training only with CPU servers. Concretely, Dorylus is 1.22x faster and 4.83x cheaper than GPU servers for massive sparse graphs. Dorylus is up to 3.8x faster and 10.7x cheaper compared to existing sampling-based systems.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 38 of its 110 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

38 of 110 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Graph Attention Networkscited by 8,923
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  7. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  8. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  11. 2015Gated Graph Sequence Neural Networkscited by 796
  12. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
26 more
  1. 2015Image-based Recommendations on Styles and Substitutescited by 706
  2. 2019Graph Transformer Networkscited by 523
  3. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  4. 2017Residual Gated Graph ConvNetscited by 346
  5. 2018Session-based Recommendation with Graph Neural Networkscited by 312
  6. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  7. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  8. 2019Hyperbolic Graph Neural Networkscited by 244
  9. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 242
  10. 2017Learning to Represent Programs with Graphscited by 220
  11. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  12. 2018GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphscited by 165
  13. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 153
  14. 2016DeepCoder: Learning to Write Programscited by 149
  15. 2017Revisiting Distributed Synchronous SGDcited by 142
  16. 2017Cross-Sentence N-ary Relation Extraction with Graph LSTMscited by 101
  17. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  18. 2017Device Placement Optimization with Reinforcement Learningcited by 86
  19. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  20. 2020Dynamic Tensor Rematerializationcited by 46
  21. 2018Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmarkcited by 21
  22. 2017Occupy the Cloud: Distributed Computing for the 99%cited by 17
  23. 2018Serverless Computing: One Step Forward, Two Steps Backcited by 12
  24. 2014GraphLab: A New Framework For Parallel Machine Learningcited by 8
  25. 2014FlashGraph: Processing Billion-Node Graphs on an Array of Commodity SSDscited by 5
  26. 2014Pregelix: Big(ger) Graph Analytics on A Dataflow Enginecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 15; 2023: 14; 2024: 11; 2025: 6; 2026: 2.

51 papers.

Sort:

  1. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  2. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  3. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  4. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  5. 2022PaSca: a Graph Neural Architecture Search System under the Scalable Paradigmcited by 37
  6. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  7. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  8. 2022Algorithm and System Co-design for Efficient Subgraph-based Graph Representation Learningcited by 28
  9. 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
  10. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  11. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  12. 2023The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Surveycited by 14
38 more
  1. 2022GNN at the Edge: Cost-Efficient Graph Neural Network Processing over Distributed Edge Serverscited by 11
  2. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  3. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  4. 2023Comprehensive Evaluation of GNN Training Systems: A Data Management Perspectivecited by 9
  5. 2021Bridging Data Center AI Systems with Edge Computing for Actionable Information Retrievalcited by 7
  6. 2021GNNear: Accelerating Full-Batch Training of Graph Neural Networks with Near-Memory Processingcited by 6
  7. 2023GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelismcited by 6
  8. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  9. 2023PiPAD: Pipelined and Parallel Dynamic GNN Training on GPUscited by 4
  10. 2023FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Trainingcited by 4
  11. 2023GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelismcited by 4
  12. 2023An Experimental Comparison of Partitioning Strategies for Distributed Graph Neural Network Trainingcited by 4
  13. 2022Distributed Graph Neural Network Training with Periodic Stale Representation Synchronizationcited by 3
  14. 2023Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communicationcited by 3
  15. 2024PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architecturescited by 3
  16. 2024Edge Graph Intelligence: Reciprocally Empowering Edge Networks with Graph Intelligencecited by 3
  17. 2024FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scalecited by 3
  18. 2025Sparsity-Aware Communication for Distributed Graph Neural Network Trainingcited by 3
  19. 2022DGI: Easy and Efficient Inference for GNNscited by 2
  20. 2022Scalable Graph Convolutional Network Training on Distributed-Memory Systemscited by 2
  21. 2023Serving Graph Neural Networks With Distributed Fog Servers For Smart IoT Servicescited by 2
  22. 2024Graph Neural Networks Gone Hogwildcited by 2
  23. 2024CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reductioncited by 2
  24. 2025Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputationcited by 2
  25. 2023DGC: Training Dynamic Graphs with Spatio-Temporal Non-Uniformity using Graph Partitioning by Chunkscited by 1
  26. 2024Single-GPU GNN Systems: Traps and Pitfallscited by 1
  27. 2024Reducing Memory Contention and I/O Congestion for Disk-based GNN Trainingcited by 1
  28. 2024Characterizing and Understanding HGNN Training on GPUscited by 1
  29. 2025Deal: Distributed End-to-End GNN Inference for All Nodescited by 1
  30. 2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1
  31. 2022Analysis and Optimization of GNN-Based Recommender Systems on Persistent Memorycited by 0
  32. 2022DistGNN-MB: Distributed Large-Scale Graph Neural Network Training on x86 via Minibatch Samplingcited by 0
  33. 2023Scalable Neural Network Training over Distributed Graphscited by 0
  34. 2024SDT-GNN: Streaming-based Distributed Training Framework for Graph Neural Networkscited by 0
  35. 2024Slicing Input Features to Accelerate Deep Learning: A Case Study with Graph Neural Networkscited by 0
  36. 2025MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Acceleratorscited by 0
  37. 2025RapidGNN: Energy and Communication-Efficient Distributed Training on Large-Scale Graph Neural Networkscited by 0
  38. 2026MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Trainingcited by 0

50 most cited of 51 All 51 (JSON)