QuaRL: Quantization for Fast and Environmentally Sustainable Reinforcement Learning

Deep reinforcement learning continues to show tremendous potential in achieving task-level autonomy, however, its computational and energy demands remain prohibitively high. In this paper, we tackle this problem by applying quantization to reinforcement learning. To that end, we introduce a novel Reinforcement Learning (RL) training paradigm, textitActorQ, to speed up actor-learner distributed RL training. textitActorQ leverages 8-bit quantized actors to speed up data collection without affecting learning convergence. Our quantized distributed RL training system, textitActorQ, demonstrates end-to-end speedups bluebetween 1.5 × and 5.41×, and faster convergence over full precision training on a range of tasks (Deepmind Control Suite) and different RL algorithms (D4PG, DQN). Furthermore, we compare the carbon emissions (Kgs of CO2) of textitActorQ versus standard reinforcement learning bluealgorithms on various tasks. Across various settings, we show that textitActorQ enables more environmentally friendly reinforcement learning by achieving bluecarbon emission improvements between 1.9× and 3.76× compared to training RL-agents in full-precision. We believe that this is the first of many future works on enabling computationally energy-efficient and sustainable reinforcement learning. The source code is available here for the public to use: urlhttps://github.com/harvard-edge/QuaRL.