mlsanityData to 14 Oct 2026

RAMP: A Flat Nanosecond Optical Network and MPI Operations for Distributed Deep Learning Systems

arXiv 2211.15226 ↗First posted 28 Nov 2022cs.DC cs.LG cs.NIcited by 1

Not on the map. Why?

Abstract (arXiv)

Distributed deep learning (DDL) systems strongly depend on network performance. Current electronic packet switched (EPS) network architectures and technologies suffer from variable diameter topologies, low-bisection bandwidth and over-subscription affecting completion time of communication and collective operations. We introduce a near-exascale, full-bisection bandwidth, all-to-all, single-hop, all-optical network architecture with nanosecond reconfiguration called RAMP, which supports large-scale distributed and parallel computing systems (12.8~Tbps per node for up to 65,536 nodes). For the first time, a custom RAMP-x MPI strategy and a network transcoder is proposed to run MPI collective operations across the optical circuit switched (OCS) network in a schedule-less and contention-less manner. RAMP achieves 7.6-171× speed-up in completion time across all MPI operations compared to realistic EPS and OCS counterparts. It can also deliver a 1.3-16× and 7.8-58× reduction in Megatron and DLRM training time respectively while offering 42-53× and 3.3-12.4× improvement in energy consumption and cost respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 75 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

15 of 75 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  10. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  11. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  12. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
3 more
  1. 2020The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelismcited by 11
  2. 2020Optimal Control of SOAs with Artificial Intelligence for Sub-Nanosecond Optical Switchingcited by 3
  3. 2020PULSE: Optical circuit switched Data Center architecture operating at nanosecond timescalescited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OptINC: Optical In-Network-Computing for Scalable Distributed Learning
  2. 2026Scheduling Parallel Optical Circuit Switches for AI Training
  3. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training
  4. 2022Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing
  5. 2024Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment
  6. 2025Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach
  7. 2026A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network
  8. 2024TeMPO: Efficient Time-Multiplexed Dynamic Photonic Tensor Core for Edge AI with Compact Slow-Light Electro-Optic Modulator