# Assurance for Deployed Continual Learning Systems

arXiv: https://arxiv.org/abs/2311.10787 · HTML: https://mlsanity.com/paper/2311.10787/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 7 of 7)

- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks](https://mlsanity.com/paper/1902.08722/) (2019)
- [Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent](https://mlsanity.com/paper/2006.11942/) (2020)
- [Training verified learners with learned verifiers](https://mlsanity.com/paper/1805.10265/) (2018)
- [Synthesis of Control Barrier Functions Using a Supervised Machine Learning Approach](https://mlsanity.com/paper/2003.04950/) (2020)
- [Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities](https://mlsanity.com/paper/1804.05892/) (2018)
- [Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications](https://mlsanity.com/paper/2103.16910/) (2021)

## Abstract (arXiv)

The future success of the Navy will depend, in part, on artificial intelligence. In practice, many artificially intelligent algorithms, and in particular deep learning models, rely on continual learning to maintain performance in dynamic environments. The software requires adaptation to maintain its initial level of performance in unseen situations. However, if not monitored properly, continual learning may lead to several issues including catastrophic forgetting in which a trained model forgets previously learned tasks when being retrained on new data. The authors created a new framework for safely performing continual learning with the goal of pairing this safety framework with a deep learning computer vision algorithm to allow for safe and high-performing automatic deck tracking on carriers and amphibious assault ships. The safety framework includes several features, such as an ensemble of convolutional neural networks to perform image classification, a manager to record confidences and determine the best answer from the ensemble, a model of the environment to predict when the system may fail to meet minimum performance metrics, a performance monitor to log system and domain performance and check against requirements, and a retraining component to update the ensemble and manager to maintain performance. The authors validated the proposed method using extensive simulation studies based on dynamic image classification. The authors showed the safety framework could probabilistically detect out of distribution data. The results also show the framework can detect when the system is no longer performing safely and can significantly extend the working envelope of an image classifier.