mlsanityData to 14 Oct 2026

MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoning

arXiv 2608.17722 ↗First posted 18 Aug 2026cs.CR cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision-Language models (VLMs) achieve outstanding performance largely due to the amount of training data available on the internet. At the same time, data holders (e.g., artists) urgently need to determine whether their data has been used for model training without authorization, which concerns both intellectual property rights and personal privacy. Data auditing, particularly through membership inference (MI), has attracted attention as a direct tool. This work proposes MemCatalyst, a set of data poisoning tools, aiming to amplify the data auditing performance on VLMs. MemCatalyst employs two strategies: Poisoning Text (PT) and Poisoning Image (PI). MemCatalyst forces VLMs to over-learn specific inconsistencies between image features and textual semantics during training, thereby increasing their susceptibility to membership information auditing. Crucially, the transferability of poisoned samples across different VLM architectures is demonstrated to be effective in the black-box setting. Extensive evaluations using five state-of-the-art data audits on two prominent VLMs demonstrate that MemCatalyst markedly enhances MI AUC scores with a minimal budget of poisoned samples, while maintaining a negligible impact on model performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 67 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

51 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabeling27 Jun
  2. 2025Membership Inference Attacks Against Vision-Language Models27 Jan
  3. 2024Membership Inference Attacks against Large Vision-Language Models5 Nov
  4. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models16 Oct
  5. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models5 Feb

Other references · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2016Wide Residual Networkscited by 3,070
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
34 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  3. 2021Membership Inference Attacks From First Principlescited by 878
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  7. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  8. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
  9. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  10. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  11. 2020Dynamic Backdoor Attacks Against Machine Learning Modelscited by 151
  12. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 141
  13. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  14. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 123
  15. 2023On the Exploitability of Instruction Tuningcited by 116
  16. 2020Graph Backdoorcited by 112
  17. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 101
  18. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  19. 2023Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Studycited by 38
  20. 2023Black-box Membership Inference Attacks against Fine-tuned Diffusion Modelscited by 35
  21. 2022M^4I: Multi-modal Models Membership Inferencecited by 34
  22. 2022Amplifying Membership Exposure via Data Poisoningcited by 34
  23. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  24. 2024Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Modelscited by 30
  25. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  26. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  27. 2024LUMIA: Linear probing for Unimodal and MultiModal Membership Inference Attacks leveraging internal LLM statescited by 10
  28. 2025Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLMcited by 10
  29. 2026VidLeaks: Membership Inference Attacks Against Text-to-Video Modelscited by 5
  30. 2024Anonymity Unveiled: A Practical Framework for Auditing Data Use in Deep Learning Modelscited by 4
  31. 2026Are LLM-Enhanced Graph Neural Networks Robust against Poisoning Attacks?cited by 4
  32. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  33. 2025When and Where do Data Poisons Attack Textual Inversion?cited by 2
  34. 2026VLALeaks: Membership Inference Attacks against Vision-Language-Action Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Data Auditing in Large Vision-Language Models
  2. 2025Instance-Level Data-Use Auditing of Visual ML Models
  3. 2024Clean Evaluations on Contaminated Visual Language Models
  4. 2026Evian: Towards Explainable Visual Instruction-tuning Data Auditing
  5. 2026To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
  6. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing
  7. 2026Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignment
  8. 2025OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models