# HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments

arXiv: https://arxiv.org/abs/2111.10635 · HTML: https://mlsanity.com/paper/2111.10635/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 83 references that are in the map.
- First posted on arXiv 20 Nov 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [DAPPLE: A Pipelined Data Parallel Approach for Training Large Models](https://mlsanity.com/paper/2007.01045/) (2020)
- [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://mlsanity.com/paper/2006.15704/) (2020)
- [A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning](https://mlsanity.com/paper/1609.04453/) (2016)
- [Device Placement Optimization with Reinforcement Learning](https://mlsanity.com/paper/1706.04972/) (2017)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [From Distributed Machine Learning to Federated Learning: A Survey](https://mlsanity.com/paper/2104.14362/) (2021)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters](https://mlsanity.com/paper/1706.03292/) (2017)
- [Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads](https://mlsanity.com/paper/2008.09213/) (2020)
- [Device Scheduling with Fast Convergence for Wireless Federated Learning](https://mlsanity.com/paper/1911.00856/) (2019)
- [HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism](https://mlsanity.com/paper/2005.14038/) (2020)
- [Recurrent Orthogonal Networks and Long-Memory Tasks](https://mlsanity.com/paper/1602.06662/) (2016)
- [Local policy search with Bayesian optimization](https://mlsanity.com/paper/2106.11899/) (2021)
- [Deep CTR Prediction in Display Advertising](https://mlsanity.com/paper/1609.06018/) (2016)
- [Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks](https://mlsanity.com/paper/1808.02621/) (2018)
- [Pipeline Parallelism for Inference on Heterogeneous Edge Computing](https://mlsanity.com/paper/2110.14895/) (2021)
- [Scheduling Algorithms for Efficient Execution of Stream Workflow Applications in Multicloud Environments](https://mlsanity.com/paper/1912.08392/) (2019)
- [MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning](https://mlsanity.com/paper/1801.03855/) (2018)
- [Toward Efficient Online Scheduling for Distributed Machine Learning Systems](https://mlsanity.com/paper/2108.02917/) (2021)

## Built on by (most cited first, 15 of 15)

- [Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization](https://mlsanity.com/paper/2310.15080/) (2023)
- [Multi-Job Intelligent Scheduling with Cross-Device Federated Learning](https://mlsanity.com/paper/2211.13430/) (2022)
- [AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices](https://mlsanity.com/paper/2312.10935/) (2023)
- [Multi-Agent Reinforcement Learning for Adaptive Resource Orchestration in Cloud-Native Clusters](https://mlsanity.com/paper/2508.10253/) (2025)
- [Efficient Asynchronous Federated Learning with Sparsification and Quantization](https://mlsanity.com/paper/2312.15186/) (2023)
- [Enhancing Trust and Privacy in Distributed Networks: A Comprehensive Survey on Blockchain-based Federated Learning](https://mlsanity.com/paper/2403.19178/) (2024)
- [Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data](https://mlsanity.com/paper/2408.05678/) (2024)
- [Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models](https://mlsanity.com/paper/2410.00131/) (2024)
- [Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources](https://mlsanity.com/paper/2207.06667/) (2022)
- [SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2505.12435/) (2025)
- [On Mask-based Image Set Desensitization with Recognition Support](https://mlsanity.com/paper/2312.08975/) (2023)
- [Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout](https://mlsanity.com/paper/2507.10430/) (2025)
- [Optimized detection of cyber-attacks on IoT networks via hybrid deep learning models](https://mlsanity.com/paper/2502.11470/) (2025)
- [A Survey From Distributed Machine Learning to Distributed Deep Learning](https://mlsanity.com/paper/2307.05232/) (2023)
- [HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training](https://mlsanity.com/paper/2609.39350/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) exploit many layers and a large number of parameters to achieve excellent performance. The training process of DNN models generally handles large-scale input data with many sparse features, which incurs high Input/Output (IO) cost, while some layers are compute-intensive. The training process generally exploits distributed computing resources to reduce training time. In addition, heterogeneous computing resources, e.g., CPUs, GPUs of multiple types, are available for the distributed training process. Thus, the scheduling of multiple layers to diverse computing resources is critical for the training process. To efficiently train a DNN model using the heterogeneous computing resources, we propose a distributed framework, i.e., Paddle-Heterogeneous Parameter Server (Paddle-HeterPS), composed of a distributed architecture and a Reinforcement Learning (RL)-based scheduling method. The advantages of Paddle-HeterPS are three-fold compared with existing frameworks. First, Paddle-HeterPS enables efficient training process of diverse workloads with heterogeneous computing resources. Second, Paddle-HeterPS exploits an RL-based method to efficiently schedule the workload of each layer to appropriate computing resources to minimize the cost while satisfying throughput constraints. Third, Paddle-HeterPS manages data storage and data communication among distributed computing resources. We carry out extensive experiments to show that Paddle-HeterPS significantly outperforms state-of-the-art approaches in terms of throughput (14.5 times higher) and monetary cost (312.3% smaller). The codes of the framework are publicly available at: https://github.com/PaddlePaddle/Paddle.