# A survey of air combat behavior modeling using machine learning

arXiv: https://arxiv.org/abs/2404.13954 · HTML: https://mlsanity.com/paper/2404.13954/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 154 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Hierarchical Imitation and Reinforcement Learning](https://mlsanity.com/paper/1803.00590/) (2018)
- [A Survey of Behavior Trees in Robotics and AI](https://mlsanity.com/paper/2005.05842/) (2020)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat](https://mlsanity.com/paper/2212.03830/) (2022)
- [Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials](https://mlsanity.com/paper/2105.00990/) (2021)
- [Engagement Decision Support for Beyond Visual Range Air Combat](https://mlsanity.com/paper/2111.03059/) (2021)
- [Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering](https://mlsanity.com/paper/2309.11247/) (2023)
- [Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails](https://mlsanity.com/paper/1901.05599/) (2019)
- [BVR Gym: A Reinforcement Learning Environment for Beyond-Visual-Range Air Combat](https://mlsanity.com/paper/2403.17533/) (2024)
- [Learning Environment for the Air Domain (LEAD)](https://mlsanity.com/paper/2304.14423/) (2023)

## Built on by (most cited first, 3 of 3)

- [Training Environment for High Performance Reinforcement Learning](https://mlsanity.com/paper/2505.01953/) (2025)
- [Enhancing Aerial Combat Tactics through Hierarchical Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.08995/) (2025)
- [Explaining Strategic Decisions in Multi-Agent Reinforcement Learning for Aerial Combat Tactics](https://mlsanity.com/paper/2505.11311/) (2025)

## Abstract (arXiv)

With the recent advances in machine learning, creating agents that behave realistically in simulated air combat has become a growing field of interest. This survey explores the application of machine learning techniques for modeling air combat behavior, motivated by the potential to enhance simulation-based pilot training. Current simulated entities tend to lack realistic behavior, and traditional behavior modeling is labor-intensive and prone to loss of essential domain knowledge between development steps. Advancements in reinforcement learning and imitation learning algorithms have demonstrated that agents may learn complex behavior from data, which could be faster and more scalable than manual methods. Yet, making adaptive agents capable of performing tactical maneuvers and operating weapons and sensors still poses a significant challenge. The survey examines applications, behavior model types, prevalent machine learning methods, and the technical and human challenges in developing adaptive and realistically behaving agents. Another challenge is the transfer of agents from learning environments to military simulation systems and the consequent demand for standardization. Four primary recommendations are presented regarding increased emphasis on beyond-visual-range scenarios, multi-agent machine learning and cooperation, utilization of hierarchical behavior models, and initiatives for standardization and research collaboration. These recommendations aim to address current issues and guide the development of more comprehensive, adaptable, and realistic machine learning-based behavior models for air combat applications.