mlsanityData to 7 Oct 2026

TicTac: Accelerating Distributed Deep Learning with Communication Scheduling

arXiv 1803.03288 ↗First posted 8 Mar 2018cs.DC cs.LG cs.PFcited by 27

Not on the map. Why?

Abstract (arXiv)

State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity of models and input data. The iteration time in these communication-heavy systems depends on the computation time, communication time and the extent of overlap of computation and communication. In this work, we identify a shortcoming in systems with graph representation for computation, such as TensorFlow and PyTorch, that result in high variance in iteration time --- random order of received parameters across workers. We develop a system, TicTac, to improve the iteration time by fixing this issue in distributed deep learning with Parameter Servers while guaranteeing near-optimal overlap of communication and computation. TicTac identifies and enforces an order of network transfers which improves the iteration time using prioritization. Our system is implemented over TensorFlow and requires no changes to the model or developer inputs. TicTac improves the throughput by up to 37.7% in inference and 19.2% in training, while also reducing straggler effect by up to 2.3×. Our code is publicly available.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 4; 2023: 3; 2024: 4; 2025: 4.

27 papers.

Sort:

By area:

  1. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  2. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  3. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  4. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 28
  5. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  6. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  7. 2022On Optimizing the Communication of Model Parallelismcited by 19
  8. 2021Distributed Learning Systems with First-order Methodscited by 17
  9. 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
  10. 2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
  11. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  12. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 8
15 more
  1. 2025Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Trainingcited by 6
  2. 2023Enhancing Neural Training via a Correlated Dynamics Modelcited by 5
  3. 2021DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Schedulingcited by 1
  4. 2021EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networkscited by 1
  5. 2021Automatic Configuration for Optimal Communication Scheduling in DNN Trainingcited by 1
  6. 2022ByteComp: Revisiting Gradient Compression in Distributed Trainingcited by 1
  7. 2021CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Mannercited by 0
  8. 2021Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Frameworkcited by 0
  9. 2021CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensationcited by 0
  10. 2021Scheduling Optimization Techniques for Neural Network Trainingcited by 0
  11. 2023On the Burstiness of Distributed Machine Learning Trafficcited by 0
  12. 2024MLCC: A Congestion Control Technique to Accelerate ML Trainingcited by 0
  13. 2024FedEx: Expediting Federated Learning over Heterogeneous Mobile Devices by Overlapping and Participant Selectioncited by 0
  14. 2025FLStore: Efficient Federated Learning Storage for non-training workloadscited by 0
  15. 2025Routing for Large ML Modelscited by 0