mlsanityData to 14 Oct 2026

Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

arXiv 2105.00990 ↗First posted 3 May 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 23 of its 71 references that are in the map. First posted on arXiv 3 May 2021.

Builds on

23 of 71 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  11. 2016The Option-Critic Architecturecited by 448
  12. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
11 more
  1. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  2. 2018Distributed Prioritized Experience Replaycited by 269
  3. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  4. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  5. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  6. 2017Meta Learning Shared Hierarchiescited by 93
  7. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  8. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 42
  9. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  10. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  11. 2017Hindsight policy gradientscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 2.

8 papers.

  1. 2024BVR Gym: A Reinforcement Learning Environment for Beyond-Visual-Range Air Combatcited by 3
  2. 2024A survey of air combat behavior modeling using machine learningcited by 3
  3. 2025A Hierarchical Reinforcement Learning Framework for Multi-UAV Combat Using Leader-Follower Strategycited by 3
  4. 2026Agile Interception of a Flying Target using Competitive Reinforcement Learningcited by 1
  5. 2024Deep Learning Based Situation Awareness for Multiple Missiles Evasioncited by 0
  6. 2024Interpretable DRL-based Maneuver Decision of UCAV Dogfightcited by 0
  7. 2024Intercepting Unauthorized Aerial Robots in Controlled Airspace Using Reinforcement Learningcited by 0
  8. 2026Temporal Action Representation Learning for Tactical Resource Control and Subsequent Maneuver Generationcited by 0