mlsanityData to 7 Oct 2026

Instance-Level Data-Use Auditing of Visual ML Models

arXiv 2503.22413 ↗First posted 28 Mar 2025cs.CR cs.LGcited by 4

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

The growing trend of legal disputes over the unauthorized use of data in machine learning (ML) systems highlights the urgent need for reliable data-use auditing mechanisms to ensure accountability and transparency in ML. We present the first proactive, instance-level, data-use auditing method designed to enable data owners to audit the use of their individual data instances in ML models, providing more fine-grained auditing results than previous work. To do so, our research generalizes previous work integrating black-box membership inference and sequential hypothesis testing, expanding its scope of application while preserving the quantifiable and tunable false-detection rate that is its hallmark. We evaluate our method on three types of visual ML models: image classifiers, visual encoders, and vision-language models (Contrastive Language-Image Pretraining (CLIP) and Bootstrapping Language-Image Pretraining (BLIP) models). In addition, we apply our method to evaluate the performance of two state-of-the-art approximate unlearning methods. As a noteworthy second contribution, our work reveals that neither method successfully removes the influence of the unlearned data instances from image classifiers and CLIP models, even if sacrificing model utility by 10%.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 56 of its 91 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

56 of 91 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services15 Sep
  2. 2023ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning6 Sep
  3. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning22 Oct
  4. 2021Machine Unlearning of Features and Labels26 Aug

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2016Wide Residual Networkscited by 3,057
  12. 2016Deep Learning with Differential Privacycited by 2,682
38 more
  1. 2019Machine Unlearningcited by 1,020
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2021Membership Inference Attacks From First Principlescited by 839
  4. 2018Object Detection with Deep Learning: A Reviewcited by 524
  5. 2019Certified Data Removal from Machine Learning Modelscited by 512
  6. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  7. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  8. 2020Label-Only Membership Inference Attackscited by 371
  9. 2019Hidden Trigger Backdoor Attackscited by 359
  10. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  11. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  12. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  13. 2020Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Datacited by 200
  14. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
  15. 2017Machine Learning Models that Remember Too Muchcited by 165
  16. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 156
  17. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  18. 2023Low-Cost High-Power Membership Inference Attackscited by 99
  19. 2020On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shapingcited by 82
  20. 2022Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protectioncited by 80
  21. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
  22. 2020Radioactive data: tracing through trainingcited by 67
  23. 2018On the Suitability of Lp-norms for Creating and Preventing Adversarial Examplescited by 57
  24. 2022Black-box Dataset Ownership Verification via Backdoor Watermarkingcited by 57
  25. 2023Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Handcited by 52
  26. 2020Towards Probabilistic Verification of Machine Unlearningcited by 46
  27. 2024Proving membership in LLM pretraining data via data watermarkscited by 38
  28. 2023Did You Train on My Dataset? Towards Public Dataset Protection with Clean-Label Backdoor Watermarkingcited by 37
  29. 2023Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Studycited by 37
  30. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  31. 2015An Exploration of Softmax Alternatives Belonging to the Spherical Loss Familycited by 27
  32. 2020Confidence sequences for sampling without replacementcited by 23
  33. 2024Towards Reliable Verification of Unauthorized Data Usage in Personalized Text-to-Image Diffusion Modelscited by 19
  34. 2024Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attackscited by 17
  35. 2024A General Framework for Data-Use Auditing of ML Modelscited by 15
  36. 2024SoK: Dataset Copyright Auditing in Machine Learning Systemscited by 14
  37. 2022Data Isotopes for Data Provenance in DNNscited by 11
  38. 2024A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspectivecited by 8
  2. 2026Fine-Grained Traceability for Transparent ML Pipelinescited by 1
  3. 2026Auditing Training Data in Generative Music Models via Black-Box Membership Inferencecited by 0
  4. 2026The Null Is the Hard Part: Exact Tests for Memorization in Generative Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Data Auditing in Large Vision-Language Models
  2. 2026MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoning
  3. 2026Evian: Towards Explainable Visual Instruction-tuning Data Auditing
  4. 2024Membership Inference Attacks against Large Vision-Language Models
  5. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models
  6. 2024Clean Evaluations on Contaminated Visual Language Models
  7. 2026VISTA: Auditing Semantic Divergence in Vision-Language Models
  8. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing