mlsanityData to 7 Oct 2026

Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping

arXiv 2409.15241 ↗First posted 23 Sep 2024cs.DC cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 31 of its 70 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

31 of 70 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
19 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  8. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  9. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  10. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  11. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  12. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  13. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  14. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 34
  15. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  16. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  17. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
  18. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  19. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operationscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 9
  2. 2025Tensor-Parallelism with Partially Synchronized Activationscited by 4
  3. 2026Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Trainingcited by 3
  4. 2025Design Space Exploration of DMA based Finer-Grain Compute Communication Overlapcited by 1
  5. 2026CUCo: An Agentic Framework for Compute and Communication Co-designcited by 1
  6. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Expertscited by 1
  7. 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communicationcited by 0
  8. 2026Affinity-Aware Sharding for Delayed Tensor Parallelismcited by 0