# TicTac: Accelerating Distributed Deep Learning with Communication Scheduling

arXiv: https://arxiv.org/abs/1803.03288 · HTML: https://mlsanity.com/paper/1803.03288/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2018.

## Built on by (most cited first, 27 of 27)

- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production](https://mlsanity.com/paper/2505.11432/) (2025)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezing](https://mlsanity.com/paper/2201.06227/) (2022)
- [On Optimizing the Communication of Model Parallelism](https://mlsanity.com/paper/2211.05322/) (2022)
- [Distributed Learning Systems with First-order Methods](https://mlsanity.com/paper/2104.05245/) (2021)
- [Towards Demystifying Serverless Machine Learning Training](https://mlsanity.com/paper/2105.07806/) (2021)
- [OneFlow: Redesign the Distributed Deep Learning Framework from Scratch](https://mlsanity.com/paper/2110.15032/) (2021)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)
- [Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping](https://mlsanity.com/paper/2409.15241/) (2024)
- [Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training](https://mlsanity.com/paper/2504.09307/) (2025)
- [Enhancing Neural Training via a Correlated Dynamics Model](https://mlsanity.com/paper/2312.13247/) (2023)
- [DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling](https://mlsanity.com/paper/2101.07968/) (2021)
- [EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks](https://mlsanity.com/paper/2110.09132/) (2021)
- [Automatic Configuration for Optimal Communication Scheduling in DNN Training](https://mlsanity.com/paper/2112.13509/) (2021)
- [ByteComp: Revisiting Gradient Compression in Distributed Training](https://mlsanity.com/paper/2205.14465/) (2022)
- [CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Manner](https://mlsanity.com/paper/2103.07974/) (2021)
- [Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework](https://mlsanity.com/paper/2105.09837/) (2021)
- [CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation](https://mlsanity.com/paper/2106.10796/) (2021)
- [Scheduling Optimization Techniques for Neural Network Training](https://mlsanity.com/paper/2110.00929/) (2021)
- [On the Burstiness of Distributed Machine Learning Traffic](https://mlsanity.com/paper/2401.00329/) (2023)
- [MLCC: A Congestion Control Technique to Accelerate ML Training](https://mlsanity.com/paper/2402.09589/) (2024)
- [FedEx: Expediting Federated Learning over Heterogeneous Mobile Devices by Overlapping and Participant Selection](https://mlsanity.com/paper/2407.00943/) (2024)
- [FLStore: Efficient Federated Learning Storage for non-training workloads](https://mlsanity.com/paper/2503.00323/) (2025)
- [Routing for Large ML Models](https://mlsanity.com/paper/2503.05324/) (2025)

## Abstract (arXiv)

State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity of models and input data. The iteration time in these communication-heavy systems depends on the computation time, communication time and the extent of overlap of computation and communication. In this work, we identify a shortcoming in systems with graph representation for computation, such as TensorFlow and PyTorch, that result in high variance in iteration time --- random order of received parameters across workers. We develop a system, TicTac, to improve the iteration time by fixing this issue in distributed deep learning with Parameter Servers while guaranteeing near-optimal overlap of communication and computation. TicTac identifies and enforces an order of network transfers which improves the iteration time using prioritization. Our system is implemented over TensorFlow and requires no changes to the model or developer inputs. TicTac improves the throughput by up to 37.7% in inference and 19.2% in training, while also reducing straggler effect by up to 2.3×. Our code is publicly available.