# A Review of Nine Physics Engines for Reinforcement Learning Research

arXiv: https://arxiv.org/abs/2407.08590 · HTML: https://mlsanity.com/paper/2407.08590/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 33 of its 67 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 33)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Emergent Complexity via Multi-Agent Competition](https://mlsanity.com/paper/1710.03748/) (2017)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control](https://mlsanity.com/paper/2103.02142/) (2021)
- [DeepMind Lab](https://mlsanity.com/paper/1612.03801/) (2016)
- [VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning](https://mlsanity.com/paper/2207.03530/) (2022)
- [EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine](https://mlsanity.com/paper/2206.10558/) (2022)
- [Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning](https://mlsanity.com/paper/1902.01378/) (2019)
- [Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds](https://mlsanity.com/paper/2210.13417/) (2022)
- [An Introduction to Multi-Agent Reinforcement Learning and Review of its Application to Autonomous Mobility](https://mlsanity.com/paper/2203.07676/) (2022)
- [Using Unity to Help Solve Intelligence](https://mlsanity.com/paper/2011.09294/) (2020)
- [Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning](https://mlsanity.com/paper/1911.01715/) (2019)
- [gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebo](https://mlsanity.com/paper/1903.06278/) (2019)
- [Godot Reinforcement Learning Agents](https://mlsanity.com/paper/2112.03636/) (2021)

## Built on by (most cited first, 12 of 12)

- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [Impact of Static Friction on Sim2Real in Robotic Reinforcement Learning](https://mlsanity.com/paper/2503.01255/) (2025)
- [Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research](https://mlsanity.com/paper/2506.12312/) (2025)
- [Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds](https://mlsanity.com/paper/2607.18135/) (2026)
- [Reinforcement Learning for Ballbot Navigation in Uneven Terrain](https://mlsanity.com/paper/2505.18417/) (2025)
- [Achieving Precise and Reliable Locomotion with Differentiable Simulation-Based System Identification](https://mlsanity.com/paper/2508.04696/) (2025)
- [CiRL: Open-Source Environments for Reinforcement Learning in Circular Economy and Net Zero](https://mlsanity.com/paper/2505.21536/) (2025)
- [VirtualEnv: A Platform for Embodied AI Research](https://mlsanity.com/paper/2601.07553/) (2026)
- [Beyond Specialization: Robust Reinforcement Learning Navigation via Procedural Map Generators](https://mlsanity.com/paper/2605.02528/) (2026)
- [Enhancing RL Generalizability in Robotics through SHAP Analysis of Algorithms and Hyperparameters](https://mlsanity.com/paper/2605.02867/) (2026)
- [Co-Design of Aeroelastic Systems with Deep Reinforcement Learning](https://mlsanity.com/paper/2607.15329/) (2026)

## Abstract (arXiv)

We present a review of popular simulation engines and frameworks used in reinforcement learning (RL) research, aiming to guide researchers in selecting tools for creating simulated physical environments for RL and training setups. It evaluates nine frameworks (Brax, Chrono, Gazebo, MuJoCo, ODE, PhysX, PyBullet, Webots, and Unity) based on their popularity, feature range, quality, usability, and RL capabilities. We highlight the challenges in selecting and utilizing physics engines for RL research, including the need for detailed comparisons and an understanding of each framework's capabilities. Key findings indicate MuJoCo as the leading framework due to its performance and flexibility, despite usability challenges. Unity is noted for its ease of use but lacks scalability and simulation fidelity. The study calls for further development to improve simulation engines' usability and performance and stresses the importance of transparency and reproducibility in RL research. This review contributes to the RL community by offering insights into the selection process for simulation engines, facilitating informed decision-making.