mlsanityData to 7 Oct 2026

EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine

arXiv 2206.10558 ↗First posted 21 Jun 2022cs.LG cs.AI cs.DCcited by 58

Not on the map. Why?

Abstract (arXiv)

There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others, aim to improve the system's overall throughput. In this paper, we aim to address a common bottleneck in the RL training system, i.e., parallel environment execution, which is often the slowest part of the whole system but receives little attention. With a curated design for paralleling RL environments, we have improved the RL environment simulation speed across different hardware setups, ranging from a laptop and a modest workstation, to a high-end machine such as NVIDIA DGX-A100. On a high-end machine, EnvPool achieves one million frames per second for the environment execution on Atari environments and three million frames per second on MuJoCo environments. When running EnvPool on a laptop, the speed is 2.8x that of the Python subprocess. Moreover, great compatibility with existing RL training libraries has been demonstrated in the open-sourced community, including CleanRL, rl_games, DeepMind Acme, etc. Finally, EnvPool allows researchers to iterate their ideas at a much faster pace and has great potential to become the de facto RL environment execution engine. Example runs show that it only takes five minutes to train agents to play Atari Pong and MuJoCo Ant on a laptop. EnvPool is open-sourced at https://github.com/sail-sg/envpool.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 17 of its 46 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

17 of 46 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
5 more
  1. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  2. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  3. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84
  4. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  5. 2021WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPUcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 15; 2024: 18; 2025: 10; 2026: 11.

58 papers.

Sort:

By area:

  1. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  3. 2024Simplifying Deep Temporal Difference Learningcited by 71
  4. 2022VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learningcited by 63
  5. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  6. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  7. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 52
  8. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 45
  9. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 35
  10. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 34
  11. 2024PufferLib: Making Reinforcement Learning Libraries and Environments Play Nicecited by 22
  12. 2024Learning Humanoid Locomotion over Challenging Terraincited by 20
38 more
  1. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  2. 2024Planning in a recurrent neural network that plays Sokobancited by 17
  3. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  4. 2024Sample-Efficient Alignment for LLMscited by 16
  5. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  6. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  7. 2024A Review of Nine Physics Engines for Reinforcement Learning Researchcited by 9
  8. 2023Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platformcited by 7
  9. 2023RL-X: A Deep Reinforcement Learning Library (not only) for RoboCupcited by 7
  10. 2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 6
  11. 2022Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learningcited by 5
  12. 2023GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Modelscited by 5
  13. 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimizationcited by 5
  14. 2025Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAXcited by 5
  15. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  16. 2023RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Controlcited by 4
  17. 2023HyperPPO: A scalable method for finding small policies for robotic controlcited by 4
  18. 2024Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimizationcited by 4
  19. 2024On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networkscited by 4
  20. 2023RLLTE: Long-Term Evolution Project of Reinforcement Learningcited by 3
  21. 2023High Throughput Training of Deep Surrogates from Large Ensemble Runscited by 3
  22. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  23. 2022Actively Learning Costly Reward Functions for Reinforcement Learningcited by 2
  24. 2023Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learningcited by 2
  25. 2023Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversitycited by 2
  26. 2024Behavioural Cloning in VizDoomcited by 2
  27. 2024ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyzecited by 2
  28. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimationcited by 2
  29. 2024Speeding up Policy Simulation in Supply Chain RLcited by 1
  30. 2025Adaptive Data Exploitation in Deep Reinforcement Learningcited by 1
  31. 2025HEPPO-GAE: Hardware-Efficient Proximal Policy Optimization with Generalized Advantage Estimationcited by 1
  32. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  33. 2026Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learningcited by 1
  34. 2026Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxycited by 1
  35. 2023Replication of Multi-agent Reinforcement Learning for the "Hide and Seek" Problemcited by 0
  36. 2024Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradientscited by 0
  37. 2025Adaptive Policy Synchronization for Scalable Reinforcement Learningcited by 0
  38. 2025ANO : Faster is Better in Noisy Landscapecited by 0

50 most cited of 58 All 58 (JSON)