# QuadSwarm: A Modular Multi-Quadrotor Simulator for Deep Reinforcement Learning with Direct Thrust Control

arXiv: https://arxiv.org/abs/2306.09537 · HTML: https://mlsanity.com/paper/2306.09537/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 13 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2006.11751/) (2020)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control](https://mlsanity.com/paper/2103.02142/) (2021)
- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [Reinforcement Learning for UAV Attitude Control](https://mlsanity.com/paper/1804.04154/) (2018)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [Neural-Swarm: Decentralized Close-Proximity Multirotor Control Using Learned Interactions](https://mlsanity.com/paper/2003.02992/) (2020)
- [Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning](https://mlsanity.com/paper/2109.07735/) (2021)
- [Air Learning: A Deep Reinforcement Learning Gym for Autonomous Aerial Robot Visual Navigation](https://mlsanity.com/paper/1906.00421/) (2019)
- [Learning Agile Flights through Narrow Gaps with Varying Angles using Onboard Sensing](https://mlsanity.com/paper/2302.11233/) (2023)

## Built on by (most cited first, 5 of 5)

- [Collision Avoidance and Navigation for a Quadrotor Swarm Using End-to-end Deep Reinforcement Learning](https://mlsanity.com/paper/2309.13285/) (2023)
- [HyperPPO: A scalable method for finding small policies for robotic control](https://mlsanity.com/paper/2309.16663/) (2023)
- [Symmetries-enhanced Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.01136/) (2025)
- [Crazyflow: An Accurate, GPU-Accelerated, Differentiable Drone Simulator in JAX](https://mlsanity.com/paper/2606.01478/) (2026)
- [FPGA-Based Neural Thrust Controller for UAVs](https://mlsanity.com/paper/2403.18703/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) has shown promise in creating robust policies for robotics tasks. However, contemporary RL algorithms are data-hungry, often requiring billions of environment transitions to train successful policies. This necessitates the use of fast and highly-parallelizable simulators. In addition to speed, such simulators need to model the physics of the robots and their interaction with the environment to a level acceptable for transferring policies learned in simulation to reality. We present QuadSwarm, a fast, reliable simulator for research in single and multi-robot RL for quadrotors that addresses both issues. QuadSwarm, with fast forward-dynamics propagation decoupled from rendering, is designed to be highly parallelizable such that throughput scales linearly with additional compute. It provides multiple components tailored toward multi-robot RL, including diverse training scenarios, and provides domain randomization to facilitate the development and sim2real transfer of multi-quadrotor control policies. Initial experiments suggest that QuadSwarm achieves over 48,500 simulation samples per second (SPS) on a single quadrotor and over 62,000 SPS on eight quadrotors on a 16-core CPU. The code can be found in https://github.com/Zhehui-Huang/quad-swarm-rl.