# Device Placement Optimization with Reinforcement Learning

arXiv: https://arxiv.org/abs/1706.04972 · HTML: https://mlsanity.com/paper/1706.04972/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2017.

## Built on by (most cited first, 30 of 86)

- [Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization](https://mlsanity.com/paper/2205.13209/) (2022)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Learning Collaborative Policies to Solve NP-hard Routing Problems](https://mlsanity.com/paper/2110.13987/) (2021)
- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [NetLLM: Adapting Large Language Models for Networking](https://mlsanity.com/paper/2402.02338/) (2024)
- [Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback](https://mlsanity.com/paper/1707.07402/) (2017)
- [Barbarians at the Gate: How AI is Upending Systems Research](https://mlsanity.com/paper/2510.06189/) (2025)
- [A Survey of Machine Learning for Computer Architecture and Systems](https://mlsanity.com/paper/2102.07952/) (2021)
- [Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads](https://mlsanity.com/paper/2105.11118/) (2021)
- [A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Cost](https://mlsanity.com/paper/2101.00494/) (2021)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning](https://mlsanity.com/paper/2201.12023/) (2022)
- [HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments](https://mlsanity.com/paper/2111.10635/) (2021)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness](https://mlsanity.com/paper/2210.07297/) (2022)
- [Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning](https://mlsanity.com/paper/2210.08238/) (2022)
- [Automap: Towards Ergonomic Automated Parallelism for ML Models](https://mlsanity.com/paper/2112.02958/) (2021)
- [Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces](https://mlsanity.com/paper/2410.15625/) (2024)
- [HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation](https://mlsanity.com/paper/2105.01585/) (2021)
- [GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism](https://mlsanity.com/paper/2406.17145/) (2024)
- [LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations](https://mlsanity.com/paper/1808.07903/) (2018)
- [Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decomposition](https://mlsanity.com/paper/2410.07574/) (2024)
- [Extract-Transform-Load for Video Streams](https://mlsanity.com/paper/2310.04830/) (2023)
- [BoGraph: Structured Bayesian Optimization From Logs for Expensive Systems with Many Parameters](https://mlsanity.com/paper/2112.08774/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [CompilerDream: Learning a Compiler World Model for General Code Optimization](https://mlsanity.com/paper/2404.16077/) (2024)
- [RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads](https://mlsanity.com/paper/2102.04285/) (2021)
- [End-to-end Adaptive Distributed Training on PaddlePaddle](https://mlsanity.com/paper/2112.02752/) (2021)

## Abstract (arXiv)

The past few years have witnessed a growth in size and computational requirements for training and inference with neural networks. Currently, a common approach to address these requirements is to use a heterogeneous distributed environment with a mixture of hardware devices such as CPUs and GPUs. Importantly, the decision of placing parts of the neural models on devices is often made by human experts based on simple heuristics and intuitions. In this paper, we propose a method which learns to optimize device placement for TensorFlow computational graphs. Key to our method is the use of a sequence-to-sequence model to predict which subsets of operations in a TensorFlow graph should run on which of the available devices. The execution time of the predicted placements is then used as the reward signal to optimize the parameters of the sequence-to-sequence model. Our main result is that on Inception-V3 for ImageNet classification, and on RNN LSTM, for language modeling and neural machine translation, our model finds non-trivial device placements that outperform hand-crafted heuristics and traditional algorithmic methods.