mlsanityData to 14 Oct 2026

Towards Reliable Testing of Machine Unlearning

arXiv 2604.16536 ↗First posted 16 Apr 2026cs.SE cs.AI cs.LGcited by 0

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine learning components are now central to AI-infused software systems, from recommendations and code assistants to clinical decision support. As regulations and governance frameworks increasingly require deleting sensitive data from deployed models, machine unlearning is emerging as a practical alternative to full retraining. However, unlearning introduces a software quality-assurance challenge: under realistic deployment constraints and imperfect oracles, how can we test that a model no longer relies on targeted information? This paper frames unlearning testing as a first-class software engineering problem. We argue that practical unlearning tests must provide (i) thorough coverage over proxy and mediated influence pathways, (ii) debuggable diagnostics that localize where leakage persists, (iii) cost-effective regression-style execution under query budgets, and (iv) black-box applicability for API-deployed models. We outline a causal, pathway-centric perspective, causal fuzzing, that generates budgeted interventions to estimate residual direct and indirect effects and produce actionable "leakage reports". Proof-of-concept results illustrate that standard attribution checks can miss residual influence due to proxy pathways, cancellation effects, and subgroup masking, motivating causal testing as a promising direction for unlearning testing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 23 of its 47 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

23 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  2. 2024Reconstruction Attacks on Machine Unlearning: Simple Models are Vulnerable30 May
  3. 2022Towards Adversarial Evaluations for Inexact Machine Unlearning17 Jan

Other references · 20

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  7. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  8. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 373
  9. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
  10. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
  11. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
  12. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
8 more
  1. 2020When Machine Unlearning Jeopardizes Privacycited by 198
  2. 2018Path-Specific Counterfactual Fairnesscited by 181
  3. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  4. 2020DoWhy: An End-to-End Library for Causal Inferencecited by 95
  5. 2022Fairness Testing: A Comprehensive Survey and Analysis of Trendscited by 62
  6. 2017FairFuzz: Targeting Rare Branches to Rapidly Increase Greybox Fuzz Testing Coveragecited by 20
  7. 2021Black-Box Testing of Deep Neural Networks Through Test Case Diversitycited by 18
  8. 2025Improving Deep Learning Framework Testing with Model-Level Metamorphic Testingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CAF'E: Causal Black-Box Testing of Machine Unlearning
  2. 2026Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget
  3. 2025Probing Knowledge Holes in Unlearned LLMs
  4. 2025Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness
  5. 2025Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective
  6. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs
  7. 2025Towards Reliable Forgetting: A Survey on Machine Unlearning Verification
  8. 2024Verifying Machine Unlearning with Explainable AI