# Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering

arXiv: https://arxiv.org/abs/2309.11247 · HTML: https://mlsanity.com/paper/2309.11247/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 55 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1710.00336/) (2017)
- [A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat](https://mlsanity.com/paper/2212.03830/) (2022)

## Built on by (most cited first, 5 of 5)

- [A survey of air combat behavior modeling using machine learning](https://mlsanity.com/paper/2404.13954/) (2024)
- [Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.11474/) (2025)
- [Enhancing Aerial Combat Tactics through Hierarchical Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.08995/) (2025)
- [Explaining Strategic Decisions in Multi-Agent Reinforcement Learning for Aerial Combat Tactics](https://mlsanity.com/paper/2505.11311/) (2025)
- [DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat](https://mlsanity.com/paper/2609.11155/) (2026)

## Abstract (arXiv)

The application of artificial intelligence to simulate air-to-air combat scenarios is attracting increasing attention. To date the high-dimensional state and action spaces, the high complexity of situation information (such as imperfect and filtered information, stochasticity, incomplete knowledge about mission targets) and the nonlinear flight dynamics pose significant challenges for accurate air combat decision-making. These challenges are exacerbated when multiple heterogeneous agents are involved. We propose a hierarchical multi-agent reinforcement learning framework for air-to-air combat with multiple heterogeneous agents. In our framework, the decision-making process is divided into two stages of abstraction, where heterogeneous low-level policies control the action of individual units, and a high-level commander policy issues macro commands given the overall mission targets. Low-level policies are trained for accurate unit combat control. Their training is organized in a learning curriculum with increasingly complex training scenarios and league-based self-play. The commander policy is trained on mission targets given pre-trained low-level policies. The empirical validation advocates the advantages of our design choices.