mlsanityData to 14 Oct 2026

Membership Inference Attacks Against Vision-Language Models

arXiv 2501.18624 ↗First posted 27 Jan 2025cs.CR cs.AIcited by 29

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs), built on pre-trained vision encoders and large language models (LLMs), have shown exceptional multi-modal understanding and dialog capabilities, positioning them as catalysts for the next technological revolution. However, while most VLM research focuses on enhancing multi-modal interaction, the risks of data misuse and leakage have been largely unexplored. This prompts the need for a comprehensive investigation of such risks in VLMs. In this paper, we conduct the first analysis of misuse and leakage detection in VLMs through the lens of membership inference attack (MIA). In specific, we focus on the instruction tuning data of VLMs, which is more likely to contain sensitive or unauthorized information. To address the limitation of existing MIA methods, we introduce a novel approach that infers membership based on a set of samples and their sensitivity to temperature, a unique parameter in VLMs. Based on this, we propose four membership inference methods, each tailored to different levels of background knowledge, ultimately arriving at the most challenging scenario. Our comprehensive evaluations show that these methods can accurately determine membership status, e.g., achieving an AUC greater than 0.8 targeting a small set consisting of only 5 samples on LLaVA.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 87 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

26 of 87 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors17 May
  2. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  3. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  9. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  10. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  11. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  12. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
11 more
  1. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 90
  2. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  3. 2018Towards Demystifying Membership Inference Attackscited by 40
  4. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
  5. 2017Towards Measuring Membership Privacycited by 29
  6. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  7. 2020On the Effectiveness of Regularization Against Membership Inference Attackscited by 24
  8. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 22
  9. 2020Membership Inference Attacks and Defenses in Classification Modelscited by 16
  10. 2024Evaluating LLM-based Personal Information Extraction and Countermeasurescited by 12
  11. 2024D'ej`a Vu Memorization in Vision-Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 16.

29 papers.

Sort:

By area:

  1. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 19
  2. 2025Membership Inference Attacks on Large-Scale Models: A Surveycited by 18
  3. 2025DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspectivecited by 8
  4. 2026VidLeaks: Membership Inference Attacks Against Text-to-Video Modelscited by 5
  5. 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
  6. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 4
  7. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  8. 2025Score-based Membership Inference on Diffusion Modelscited by 3
  9. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  10. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?cited by 2
  11. 2025A Longitudinal Measurement of Privacy Policy Evolution for Large Language Modelscited by 2
  12. 2026Membership Inference Attacks against Large Audio Language Modelscited by 2
17 more
  1. 2026VLALeaks: Membership Inference Attacks against Vision-Language-Action Modelscited by 2
  2. 2025Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraintcited by 1
  3. 2025OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Modelscited by 1
  4. 2026Position: Life-Logging Video Streams Make the Privacy-Utility Trade-off Inevitablecited by 1
  5. 2025The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learningcited by 0
  6. 2026Quantifying Membership Disclosure Risk for Tabular Synthetic Data Using Kernel Density Estimatorscited by 0
  7. 2026Membership Inference Attacks Against Video Large Language Modelscited by 0
  8. 2026Membership Inference Attacks on Vision-Language-Action Modelscited by 0
  9. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbationcited by 0
  10. 2026Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignmentcited by 0
  11. 2026Provable Joint Decontamination for Benchmarking Multiple Large Language Modelscited by 0
  12. 2026Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Modelscited by 0
  13. 2026One Framework for All: Cross-Modal Membership Inference for Generative Modelscited by 0
  14. 2026When T2I Synthetic Data Backfires: Amplified Privacy Risks in Real-Synthetic Mix Trainingcited by 0
  15. 2026MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoningcited by 0
  16. 2026DIME: Query-Efficient Framework for Membership Inference on Diffusion Modelscited by 0
  17. 2026COMPASS: Membership Inference Attack Against Vision Large Language Models via Adaptive Noise Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Membership Inference Attacks against Large Vision-Language Models
  2. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models
  3. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models
  4. 2025Revisiting Data Auditing in Large Vision-Language Models
  5. 2025Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models
  6. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  7. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  8. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends