# Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control

arXiv: https://arxiv.org/abs/2103.02142 · HTML: https://mlsanity.com/paper/2103.02142/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Neural Lander: Stable Drone Landing Control using Learned Dynamics](https://mlsanity.com/paper/1811.08027/) (2018)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [An empirical investigation of the challenges of real-world reinforcement learning](https://mlsanity.com/paper/2003.11881/) (2020)
- [Reinforcement Learning for UAV Attitude Control](https://mlsanity.com/paper/1804.04154/) (2018)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [Design of a Trajectory Tracking Controller for a Nanoquadcopter](https://mlsanity.com/paper/1608.05786/) (2016)

## Built on by (most cited first, 30 of 77)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning](https://mlsanity.com/paper/2207.03530/) (2022)
- [RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning](https://mlsanity.com/paper/2504.18904/) (2025)
- [safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://mlsanity.com/paper/2109.06325/) (2021)
- [Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response](https://mlsanity.com/paper/2312.11460/) (2023)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [Learning to Fly in Seconds](https://mlsanity.com/paper/2311.13081/) (2023)
- [CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning](https://mlsanity.com/paper/2310.12972/) (2023)
- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models](https://mlsanity.com/paper/2409.19471/) (2024)
- [An Introduction to Multi-Agent Reinforcement Learning and Review of its Application to Autonomous Mobility](https://mlsanity.com/paper/2203.07676/) (2022)
- [SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum](https://mlsanity.com/paper/2412.16346/) (2024)
- [Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks](https://mlsanity.com/paper/2406.15149/) (2024)
- [Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions](https://mlsanity.com/paper/2305.09793/) (2023)
- [A Review of Nine Physics Engines for Reinforcement Learning Research](https://mlsanity.com/paper/2407.08590/) (2024)
- [Vision-based Learning for Drones: A Survey](https://mlsanity.com/paper/2312.05019/) (2023)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight](https://mlsanity.com/paper/2407.14783/) (2024)
- [Adaptive Control of SE(3) Hamiltonian Dynamics with Learned Disturbance Features](https://mlsanity.com/paper/2109.09974/) (2021)
- [Guaranteed Conformance of Neurosymbolic Models to Natural Constraints](https://mlsanity.com/paper/2212.01346/) (2022)
- [Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigation](https://mlsanity.com/paper/2302.04094/) (2023)
- [MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.02006/) (2023)
- [Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpoint](https://mlsanity.com/paper/2404.05051/) (2024)
- [Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.16720/) (2024)
- [QuadSwarm: A Modular Multi-Quadrotor Simulator for Deep Reinforcement Learning with Direct Thrust Control](https://mlsanity.com/paper/2306.09537/) (2023)
- [SwarmGPT: Combining Large Language Models with Safe Motion Planning for Drone Swarm Choreography](https://mlsanity.com/paper/2412.08428/) (2024)
- [CARLA-Air: Fly Drones Inside a CARLA World -- A Unified Infrastructure for Air-Ground Embodied Intelligence](https://mlsanity.com/paper/2603.28032/) (2026)
- [Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors](https://mlsanity.com/paper/2201.01369/) (2022)
- [Neural Solvers for Fast and Accurate Numerical Optimal Control](https://mlsanity.com/paper/2203.08072/) (2022)

## Abstract (arXiv)

Robotic simulators are crucial for academic research and education as well as the development of safety-critical applications. Reinforcement learning environments -- simple simulations coupled with a problem specification in the form of a reward function -- are also important to standardize the development (and benchmarking) of learning algorithms. Yet, full-scale simulators typically lack portability and parallelizability. Vice versa, many reinforcement learning environments trade-off realism for high sample throughputs in toy-like problems. While public data sets have greatly benefited deep learning and computer vision, we still lack the software tools to simultaneously develop -- and fairly compare -- control theory and reinforcement learning approaches. In this paper, we propose an open-source OpenAI Gym-like environment for multiple quadcopters based on the Bullet physics engine. Its multi-agent and vision based reinforcement learning interfaces, as well as the support of realistic collisions and aerodynamic effects, make it, to the best of our knowledge, a first of its kind. We demonstrate its use through several examples, either for control (trajectory tracking with PID control, multi-robot flight with downwash, etc.) or reinforcement learning (single and multi-agent stabilization tasks), hoping to inspire future research that combines control theory and machine learning.