# Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks

arXiv: https://arxiv.org/abs/1808.02621 · HTML: https://mlsanity.com/paper/1808.02621/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2018.

## Built on by (most cited first, 19 of 19)

- [HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework](https://mlsanity.com/paper/2112.07221/) (2021)
- [PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers](https://mlsanity.com/paper/2102.03161/) (2021)
- [Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning](https://mlsanity.com/paper/2201.12023/) (2022)
- [HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments](https://mlsanity.com/paper/2111.10635/) (2021)
- [ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table](https://mlsanity.com/paper/2104.08542/) (2021)
- [Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression](https://mlsanity.com/paper/2407.01378/) (2024)
- [GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Model](https://mlsanity.com/paper/2205.11048/) (2022)
- [DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce](https://mlsanity.com/paper/2602.08923/) (2026)
- [Kimad: Adaptive Gradient Compression with Bandwidth Awareness](https://mlsanity.com/paper/2312.08053/) (2023)
- [SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification](https://mlsanity.com/paper/2312.10365/) (2023)
- [A Learning Search Algorithm for the Restricted Longest Common Subsequence Problem](https://mlsanity.com/paper/2410.12031/) (2024)
- [S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning](https://mlsanity.com/paper/2110.02140/) (2021)
- [EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks](https://mlsanity.com/paper/2110.09132/) (2021)
- [Rec-AD: An Efficient Computation Framework for FDIA Detection Based on Tensor Train Decomposition and Deep Learning Recommendation Model](https://mlsanity.com/paper/2507.14668/) (2025)
- [Good Intentions: Adaptive Parameter Management via Intent Signaling](https://mlsanity.com/paper/2206.00470/) (2022)
- [Expediting Distributed DNN Training with Device Topology-Aware Graph Deployment](https://mlsanity.com/paper/2302.06126/) (2023)
- [A Survey From Distributed Machine Learning to Distributed Deep Learning](https://mlsanity.com/paper/2307.05232/) (2023)
- [Learning to Generate Parameters of ConvNets for Unseen Image Data](https://mlsanity.com/paper/2310.11862/) (2023)
- [First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training](https://mlsanity.com/paper/2510.14614/) (2025)

## Abstract (arXiv)

The employment of high-performance servers and GPU accelerators for training deep neural network models have greatly accelerated recent advances in deep learning (DL). DL frameworks, such as TensorFlow, MXNet, and Caffe2, have emerged to assist DL researchers to train their models in a distributed manner. Although current DL frameworks scale well for image classification models, there remain opportunities for scalable distributed training on natural language processing (NLP) models. We found that current frameworks show relatively low scalability on training NLP models due to the lack of consideration to the difference in sparsity of model parameters. In this paper, we propose Parallax, a framework that optimizes data parallel training by utilizing the sparsity of model parameters. Parallax introduces a hybrid approach that combines Parameter Server and AllReduce architectures to optimize the amount of data transfer according to the sparsity. Experiments show that Parallax built atop TensorFlow achieves scalable training throughput on both dense and sparse models while requiring little effort from its users. Parallax achieves up to 2.8x, 6.02x speedup for NLP models than TensorFlow and Horovod with 48 GPUs, respectively. The training speed for the image classification models is equal to Horovod and 1.53x faster than TensorFlow.