# The Architectural Implications of Distributed Reinforcement Learning on CPU-GPU Systems

arXiv: https://arxiv.org/abs/2012.04210 · HTML: https://mlsanity.com/paper/2012.04210/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 5 of 5)

- [QAPPA: Quantization-Aware Power, Performance, and Area Modeling of DNN Accelerators](https://mlsanity.com/paper/2205.08648/) (2022)
- [QADAM: Quantization-Aware DNN Accelerator Modeling for Pareto-Optimality](https://mlsanity.com/paper/2205.13045/) (2022)
- [QUIDAM: A Framework for Quantization-Aware DNN Accelerator and Model Co-Exploration](https://mlsanity.com/paper/2206.15463/) (2022)
- [Efficient Deep Learning Using Non-Volatile Memory Technology](https://mlsanity.com/paper/2206.13601/) (2022)
- [Efficient Parallel Reinforcement Learning Framework using the Reactor Model](https://mlsanity.com/paper/2312.04704/) (2023)

## Abstract (arXiv)

With deep reinforcement learning (RL) methods achieving results that exceed human capabilities in games, robotics, and simulated environments, continued scaling of RL training is crucial to its deployment in solving complex real-world problems. However, improving the performance scalability and power efficiency of RL training through understanding the architectural implications of CPU-GPU systems remains an open problem. In this work we investigate and improve the performance and power efficiency of distributed RL training on CPU-GPU systems by approaching the problem not solely from the GPU microarchitecture perspective but following a holistic system-level analysis approach. We quantify the overall hardware utilization on a state-of-the-art distributed RL training framework and empirically identify the bottlenecks caused by GPU microarchitectural, algorithmic, and system-level design choices. We show that the GPU microarchitecture itself is well-balanced for state-of-the-art RL frameworks, but further investigation reveals that the number of actors running the environment interactions and the amount of hardware resources available to them are the primary performance and power efficiency limiters. To this end, we introduce a new system design metric, CPU/GPU ratio, and show how to find the optimal balance between CPU and GPU resources when designing scalable and efficient CPU-GPU systems for RL training.