# Embedded Mean Field Reinforcement Learning for Perimeter-defense Game

arXiv: https://arxiv.org/abs/2505.14209 · HTML: https://mlsanity.com/paper/2505.14209/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning](https://mlsanity.com/paper/1710.05741/) (2017)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [Plannable Approximations to MDP Homomorphisms: Equivariance under Actions](https://mlsanity.com/paper/2002.11963/) (2020)
- [Multi Type Mean Field Reinforcement Learning](https://mlsanity.com/paper/2002.02513/) (2020)
- [Maximum Entropy Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.10715/) (2023)
- [Decision Transformers for Wireless Communications: A New Paradigm of Resource Management](https://mlsanity.com/paper/2404.05199/) (2024)
- [A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat](https://mlsanity.com/paper/2212.03830/) (2022)
- [Cooperative Team Strategies for Multi-player Perimeter-Defense Games](https://mlsanity.com/paper/1912.04342/) (2019)
- [MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.02006/) (2023)
- [Defending a Perimeter from a Ground Intruder Using an Aerial Defender: Theory and Practice](https://mlsanity.com/paper/2109.02852/) (2021)
- [Age of Information Minimization using Multi-agent UAVs based on AI-Enhanced Mean Field Resource Allocation](https://mlsanity.com/paper/2405.00056/) (2024)
- [Perimeter-defense Game between Aerial Defender and Ground Intruder](https://mlsanity.com/paper/2012.14980/) (2020)
- [Perimeter-defense Game on Arbitrary Convex Shapes](https://mlsanity.com/paper/1909.03989/) (2019)
- [The Role of Heterogeneity in Autonomous Perimeter Defense Problems](https://mlsanity.com/paper/2202.10433/) (2022)
- [Vision-based Perimeter Defense via Multiview Pose Estimation](https://mlsanity.com/paper/2209.12136/) (2022)
- [Learning Decentralized Strategies for a Perimeter Defense Game with Graph Neural Networks](https://mlsanity.com/paper/2211.01757/) (2022)
- [Dynamic Trajectory and Power Control in Ultra-Dense UAV Networks: A Mean-Field Reinforcement Learning Approach](https://mlsanity.com/paper/2411.14052/) (2024)
- [Dynamic Virtual Resource Allocation for 5G and Beyond Network Slicing](https://mlsanity.com/paper/2004.00506/) (2020)
- [Competitive Perimeter Defense on a Line](https://mlsanity.com/paper/2103.11787/) (2021)

## Abstract (arXiv)

With the rapid advancement of unmanned aerial vehicles (UAVs) and missile technologies, perimeter-defense game between attackers and defenders for the protection of critical regions have become increasingly complex and strategically significant across a wide range of domains. However, existing studies predominantly focus on small-scale, simplified two-dimensional scenarios, often overlooking realistic environmental perturbations, motion dynamics, and inherent heterogeneity--factors that pose substantial challenges to real-world applicability. To bridge this gap, we investigate large-scale heterogeneous perimeter-defense game in a three-dimensional setting, incorporating realistic elements such as motion dynamics and wind fields. We derive the Nash equilibrium strategies for both attackers and defenders, characterize the victory regions, and validate our theoretical findings through extensive simulations. To tackle large-scale heterogeneous control challenges in defense strategies, we propose an Embedded Mean-Field Actor-Critic (EMFAC) framework. EMFAC leverages representation learning to enable high-level action aggregation in a mean-field manner, supporting scalable coordination among defenders. Furthermore, we introduce a lightweight agent-level attention mechanism based on reward representation, which selectively filters observations and mean-field information to enhance decision-making efficiency and accelerate convergence in large-scale tasks. Extensive simulations across varying scales demonstrate the effectiveness and adaptability of EMFAC, which outperforms established baselines in both convergence speed and overall performance. To further validate practicality, we test EMFAC in small-scale real-world experiments and conduct detailed analyses, offering deeper insights into the framework's effectiveness in complex scenarios.