# A Conformal Predictive Measure for Assessing Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2505.10677 · HTML: https://mlsanity.com/paper/2505.10677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 15 May 2025.

## Builds on (most cited first, 13 of 13)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Toward Understanding Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/1908.01091/) (2019)
- [Training Uncertainty-Aware Classifiers with Conformalized Deep Learning](https://mlsanity.com/paper/2205.05878/) (2022)
- [Testing for concept shift online](https://mlsanity.com/paper/2012.14246/) (2020)
- [Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting](https://mlsanity.com/paper/2102.01906/) (2021)
- [Large-Scale AI in Telecom: Charting the Roadmap for Innovation, Scalability, and Enhanced Digital Experiences](https://mlsanity.com/paper/2503.04184/) (2025)
- [How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning](https://mlsanity.com/paper/2407.07668/) (2024)
- [Trustworthy Intrusion Detection: Confidence Estimation Using Latent Space](https://mlsanity.com/paper/2409.13774/) (2024)
- [A Latent Space Metric for Enhancing Prediction Confidence in Earth Observation Data](https://mlsanity.com/paper/2401.17342/) (2024)

## Abstract (arXiv)

This work introduces a novel methodology for assessing catastrophic forgetting (CF) in continual learning. We propose a new conformal prediction (CP)-based metric, termed the Conformal Prediction Confidence Factor (CPCF), to quantify and evaluate CF effectively. Our framework leverages adaptive CP to estimate forgetting by monitoring the model's confidence on previously learned tasks. This approach provides a dynamic and practical solution for monitoring and measuring CF of previous tasks as new ones are introduced, offering greater suitability for real-world applications. Experimental results on four benchmark datasets demonstrate a strong correlation between CPCF and the accuracy of previous tasks, validating the reliability and interpretability of the proposed metric. Our results highlight the potential of CPCF as a robust and effective tool for assessing and understanding CF in dynamic learning environments.