# Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance

arXiv: https://arxiv.org/abs/2105.02331 · HTML: https://mlsanity.com/paper/2105.02331/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference](https://mlsanity.com/paper/1506.02158/) (2015)
- [Data Distillation: Towards Omni-Supervised Learning](https://mlsanity.com/paper/1712.04440/) (2017)
- [Generalization and Regularization in DQN](https://mlsanity.com/paper/1810.00123/) (2018)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [Safe Reinforcement Learning with Model Uncertainty Estimates](https://mlsanity.com/paper/1810.08700/) (2018)
- [Certified Adversarial Robustness for Deep Reinforcement Learning](https://mlsanity.com/paper/1910.12908/) (2019)
- [Robust Online Monitoring of Signal Temporal Logic](https://mlsanity.com/paper/1506.08234/) (2015)
- [Calibrated Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1906.08312/) (2019)
- [Implicit Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1711.01297/) (2017)
- [Image Classification of Melanoma, Nevus and Seborrheic Keratosis by Deep Neural Network Ensemble](https://mlsanity.com/paper/1703.03108/) (2017)
- [A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance](https://mlsanity.com/paper/2003.08353/) (2020)
- [A Deep Ensemble Multi-Agent Reinforcement Learning Approach for Air Traffic Control](https://mlsanity.com/paper/2004.01387/) (2020)
- [An Autonomous Free Airspace En-route Controller using Deep Reinforcement Learning Techniques](https://mlsanity.com/paper/2007.01599/) (2020)

## Built on by (most cited first, 8 of 8)

- [Representation Uncertainty in Self-Supervised Learning as Variational Inference](https://mlsanity.com/paper/2203.11437/) (2022)
- [A Survey on Reinforcement Learning in Aviation Applications](https://mlsanity.com/paper/2211.02147/) (2022)
- [Integrated Conflict Management for UAM with Strategic Demand Capacity Balancing and Learning-based Tactical Deconfliction](https://mlsanity.com/paper/2305.10556/) (2023)
- [AAM-Gym: Artificial Intelligence Testbed for Advanced Air Mobility](https://mlsanity.com/paper/2206.04513/) (2022)
- [A Verification Framework for Certifying Learning-Based Safety-Critical Aviation Systems](https://mlsanity.com/paper/2205.04590/) (2022)
- [Towards a Standardized Reinforcement Learning Framework for AAM Contingency Management](https://mlsanity.com/paper/2311.10805/) (2023)
- [Separation Assurance between Heterogeneous Fleets of Small Unmanned Aerial Systems via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.01041/) (2026)
- [Explainable Reinforcement Learning for assisting Air Traffic Controllers](https://mlsanity.com/paper/2607.22525/) (2026)

## Abstract (arXiv)

The separation assurance task will be extremely challenging for air traffic controllers in a complex and high density airspace environment. Deep reinforcement learning (DRL) was used to develop an autonomous separation assurance framework in our previous work where the learned model advised speed maneuvers. In order to improve the safety of this model in unseen environments with uncertainties, in this work we propose a safety module for DRL in autonomous separation assurance applications. The proposed module directly addresses both model uncertainty and state uncertainty to improve safety. Our safety module consists of two sub-modules: (1) the state safety sub-module is based on the execution-time data augmentation method to introduce state disturbances in the model input state; (2) the model safety sub-module is a Monte-Carlo dropout extension that learns the posterior distribution of the DRL model policy. We demonstrate the effectiveness of the two sub-modules in an open-source air traffic simulator with challenging environment settings. Through extensive numerical experiments, our results show that the proposed sub-safety modules help the DRL agent significantly improve its safety performance in an autonomous separation assurance task.