# A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat

arXiv: https://arxiv.org/abs/2212.03830 · HTML: https://mlsanity.com/paper/2212.03830/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 57 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning](https://mlsanity.com/paper/1804.00810/) (2018)
- [UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios](https://mlsanity.com/paper/2203.14477/) (2022)
- [Enhanced Rolling Horizon Evolution Algorithm with Opponent Model Learning: Results for the Fighting Game AI Competition](https://mlsanity.com/paper/2003.13949/) (2020)
- [Empirical Policy Optimization for n-Player Markov Games](https://mlsanity.com/paper/2110.08979/) (2021)

## Built on by (most cited first, 10 of 10)

- [Cross-domain Random Pre-training with Prototypes for Reinforcement Learning](https://mlsanity.com/paper/2302.05614/) (2023)
- [TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning](https://mlsanity.com/paper/2505.19769/) (2025)
- [Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering](https://mlsanity.com/paper/2309.11247/) (2023)
- [A survey of air combat behavior modeling using machine learning](https://mlsanity.com/paper/2404.13954/) (2024)
- [Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learning](https://mlsanity.com/paper/2405.11740/) (2024)
- [A Hierarchical Reinforcement Learning Framework for Multi-UAV Combat Using Leader-Follower Strategy](https://mlsanity.com/paper/2501.13132/) (2025)
- [Interpretable DRL-based Maneuver Decision of UCAV Dogfight](https://mlsanity.com/paper/2407.01571/) (2024)
- [Rule-Based Conflict-Free Decision Framework in Swarm Confrontation](https://mlsanity.com/paper/2503.07077/) (2025)
- [Embedded Mean Field Reinforcement Learning for Perimeter-defense Game](https://mlsanity.com/paper/2505.14209/) (2025)
- [Perfecting Aircraft Maneuvers with Reinforcement Learning](https://mlsanity.com/paper/2604.24338/) (2026)

## Abstract (arXiv)

Unmanned combat air vehicle (UCAV) combat is a challenging scenario with continuous action space. In this paper, we propose a general hierarchical framework to resolve the within-vision-range (WVR) air-to-air combat problem under 6 dimensions of degree (6-DOF) dynamics. The core idea is to divide the whole decision process into two loops and use reinforcement learning (RL) to solve them separately. The outer loop takes into account the current combat situation and decides the expected macro behavior of the aircraft according to a combat strategy. Then the inner loop tracks the macro behavior with a flight controller by calculating the actual input signals for the aircraft. We design the Markov decision process for both the outer loop strategy and inner loop controller, and train them by proximal policy optimization (PPO) algorithm. For the inner loop controller, we design an effective reward function to accurately track various macro behavior. For the outer loop strategy, we further adopt a fictitious self-play mechanism to improve the combat performance by constantly combating against the historical strategies. Experiment results show that the inner loop controller can achieve better tracking performance than fine-tuned PID controller, and the outer loop strategy can perform complex maneuvers to get higher and higher winning rate, with the generation evolves.