mlsanityData to 14 Oct 2026

A Review of Nine Physics Engines for Reinforcement Learning Research

arXiv 2407.08590 ↗First posted 11 Jul 2024cs.AI cs.LG cs.MAcited by 12

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

We present a review of popular simulation engines and frameworks used in reinforcement learning (RL) research, aiming to guide researchers in selecting tools for creating simulated physical environments for RL and training setups. It evaluates nine frameworks (Brax, Chrono, Gazebo, MuJoCo, ODE, PhysX, PyBullet, Webots, and Unity) based on their popularity, feature range, quality, usability, and RL capabilities. We highlight the challenges in selecting and utilizing physics engines for RL research, including the need for detailed comparisons and an understanding of each framework's capabilities. Key findings indicate MuJoCo as the leading framework due to its performance and flexibility, despite usability challenges. Unity is noted for its ease of use but lacks scalability and simulation fidelity. The study calls for further development to improve simulation engines' usability and performance and stresses the importance of transparency and reproducibility in RL research. This review contributes to the RL community by offering insights into the selection process for simulation engines, facilitating informed decision-making.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 67 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

33 of 67 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2016OpenAI Gymcited by 1,736
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  7. 2018DeepMind Control Suitecited by 956
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  9. 2018Learning Dexterous In-Hand Manipulationcited by 687
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  12. 2018Unity: A General Platform for Intelligent Agentscited by 236
21 more
  1. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 220
  2. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  3. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  4. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  5. 2017Emergent Complexity via Multi-Agent Competitioncited by 120
  6. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  7. 2023The Semantic Scholar Open Data Platformcited by 118
  8. 2021Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Controlcited by 77
  9. 2016DeepMind Labcited by 74
  10. 2022VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learningcited by 68
  11. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  12. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  13. 2022Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worldscited by 17
  14. 2022An Introduction to Multi-Agent Reinforcement Learning and Review of its Application to Autonomous Mobilitycited by 16
  15. 2020Using Unity to Help Solve Intelligencecited by 13
  16. 2019Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learningcited by 11
  17. 2019gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebocited by 7
  18. 2021Godot Reinforcement Learning Agentscited by 4
  19. 2014Tools for dynamics simulation of robots: a survey based on user feedbackcited by 3
  20. 2022MultiRoboLearn: An open-source Framework for Multi-robot Deep Reinforcement Learningcited by 2
  21. 2020Can Reinforcement Learning for Continuous Control Generalize Across Physics Engines?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  2. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  3. 2025Impact of Static Friction on Sim2Real in Robotic Reinforcement Learningcited by 4
  4. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  5. 2026Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupedscited by 2
  6. 2025Reinforcement Learning for Ballbot Navigation in Uneven Terraincited by 1
  7. 2025Achieving Precise and Reliable Locomotion with Differentiable Simulation-Based System Identificationcited by 1
  8. 2025CiRL: Open-Source Environments for Reinforcement Learning in Circular Economy and Net Zerocited by 0
  9. 2026VirtualEnv: A Platform for Embodied AI Researchcited by 0
  10. 2026Beyond Specialization: Robust Reinforcement Learning Navigation via Procedural Map Generatorscited by 0
  11. 2026Enhancing RL Generalizability in Robotics through SHAP Analysis of Algorithms and Hyperparameterscited by 0
  12. 2026Co-Design of Aeroelastic Systems with Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimisation of the Accelerator Control by Reinforcement Learning: A Simulation-Based Approach
  2. 2025Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning
  3. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robots
  4. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  5. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
  6. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
  7. 2026The Open Ant: A Robot Platform for Reinforcement Learning Research
  8. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions