# Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games

arXiv: https://arxiv.org/abs/2506.02849 · HTML: https://mlsanity.com/paper/2506.02849/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [A Generalized Training Approach for Multiagent Learning](https://mlsanity.com/paper/1909.12823/) (2019)
- [A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight](https://mlsanity.com/paper/2202.10796/) (2022)
- [Autonomous Drone Racing: A Survey](https://mlsanity.com/paper/2301.01755/) (2023)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories](https://mlsanity.com/paper/1712.02402/) (2017)
- [Decentralized Multi-Agent Pursuit using Deep Reinforcement Learning](https://mlsanity.com/paper/2010.08193/) (2020)
- [Learning Multi-Pursuit Evasion for Safe Targeted Navigation of Drones](https://mlsanity.com/paper/2304.03443/) (2023)
- [Cooperative Pursuit with Multi-Pursuer and One Faster Free-moving Evader](https://mlsanity.com/paper/2001.04731/) (2020)
- [Multi-Target Pursuit by a Decentralized Heterogeneous UAV Swarm using Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.01799/) (2023)
- [Towards Safe Mid-Air Drone Interception: Strategies for Tracking & Capture](https://mlsanity.com/paper/2405.13542/) (2024)
- [Online Planning for Multi-UAV Pursuit-Evasion in Unknown Environments Using Deep Reinforcement Learning](https://mlsanity.com/paper/2409.15866/) (2024)
- [Variable Time-Step MPC for Agile Multi-Rotor UAV Interception of Dynamic Targets](https://mlsanity.com/paper/2503.14184/) (2025)

## Built on by (most cited first, 3 of 3)

- [Learning Agile Intruder Interception using Differentiable Quadrotor Dynamics](https://mlsanity.com/paper/2607.02472/) (2026)
- [Battery-Aware Reinforcement Learning for Aggressive Quadrotor Flight](https://mlsanity.com/paper/2609.37316/) (2026)
- [Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion](https://mlsanity.com/paper/2610.03830/) (2026)

## Abstract (arXiv)

In this letter we study 1v1 quadrotor pursuit-evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other. Such adversarial settings face well-known challenges: each agent's policy changes during training, creating a non-stationary environment; agents might overfit to the current opponent and forget earlier strategies (catastrophic forgetting); and the competitive dynamics can cause strategy cycling or policy collapse. To address these issues, we propose Asynchronous Multi-Stage Population-Based training with Hedge sampling (AMSPBH), a method based on Policy-Space Response Oracles (PSRO) and adapted to quadrotor RL control. PSRO maintains a population of previously trained policies and trains new approximate best responses against mixtures of that population instead of against a single opponent. In AMSPBH, each generation trains one agent with Proximal Policy Optimization (PPO) against frozen opponent policies, while a Hedge sampler assigns higher probability to opponents that are currently difficult to beat. We show that: (i) AMSPBH discovers new strategies while retaining competence against older opponents, reaching a regime where additional best-response training gives limited improvement; (ii) compared to training against only the latest opponent, population-based training generalizes better across diverse and unseen strategies; and (iii) the learned population contains distinct pursuit and evasion behaviors, providing useful strategic diversity for finding weaknesses and improving controller robustness. We validate the trained policies with hardware experiments on Crazyflie brushless quadrotors, showing zero-shot sim-to-real transfer of agile, reactive pursuit-evasion behavior against both handcrafted and learned adversaries, with physical flights reaching up to 4.87 m/s.