# Membership Inference Attacks Against Vision-Language Models

arXiv: https://arxiv.org/abs/2501.18624 · HTML: https://mlsanity.com/paper/2501.18624/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 87 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [Understanding Membership Inferences on Well-Generalized Learning Models](https://mlsanity.com/paper/1802.04889/) (2018)
- [VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models](https://mlsanity.com/paper/2402.13851/) (2024)
- [ImgTrojan: Jailbreaking Vision-Language Models with ONE Image](https://mlsanity.com/paper/2403.02910/) (2024)
- [A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine](https://mlsanity.com/paper/2405.08603/) (2024)
- [Towards Demystifying Membership Inference Attacks](https://mlsanity.com/paper/1807.09173/) (2018)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [Towards Measuring Membership Privacy](https://mlsanity.com/paper/1712.09136/) (2017)
- [Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors](https://mlsanity.com/paper/2405.10529/) (2024)
- [MoPe: Model Perturbation-based Privacy Attacks on Language Models](https://mlsanity.com/paper/2310.14369/) (2023)
- [On the Effectiveness of Regularization Against Membership Inference Attacks](https://mlsanity.com/paper/2006.05336/) (2020)
- [PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging](https://mlsanity.com/paper/2401.02797/) (2024)
- [Membership Inference Attacks and Defenses in Classification Models](https://mlsanity.com/paper/2002.12062/) (2020)
- [Evaluating LLM-based Personal Information Extraction and Countermeasures](https://mlsanity.com/paper/2408.07291/) (2024)
- [D'ej`a Vu Memorization in Vision-Language Models](https://mlsanity.com/paper/2402.02103/) (2024)

## Built on by (most cited first, 29 of 29)

- [Vid-SME: Membership Inference Attacks against Large Video Understanding Models](https://mlsanity.com/paper/2506.03179/) (2025)
- [Membership Inference Attacks on Large-Scale Models: A Survey](https://mlsanity.com/paper/2503.19338/) (2025)
- [DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective](https://mlsanity.com/paper/2507.05622/) (2025)
- [VidLeaks: Membership Inference Attacks Against Text-to-Video Models](https://mlsanity.com/paper/2601.11210/) (2026)
- [DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models](https://mlsanity.com/paper/2509.09097/) (2025)
- [Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing](https://mlsanity.com/paper/2511.01952/) (2025)
- [SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling](https://mlsanity.com/paper/2508.09105/) (2025)
- [Score-based Membership Inference on Diffusion Models](https://mlsanity.com/paper/2509.25003/) (2025)
- [Membership Inference Attacks on Tokenizers of Large Language Models](https://mlsanity.com/paper/2510.05699/) (2025)
- [Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?](https://mlsanity.com/paper/2511.20710/) (2025)
- [A Longitudinal Measurement of Privacy Policy Evolution for Large Language Models](https://mlsanity.com/paper/2511.21758/) (2025)
- [Membership Inference Attacks against Large Audio Language Models](https://mlsanity.com/paper/2603.28378/) (2026)
- [VLALeaks: Membership Inference Attacks against Vision-Language-Action Models](https://mlsanity.com/paper/2606.15165/) (2026)
- [Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint](https://mlsanity.com/paper/2508.20443/) (2025)
- [OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models](https://mlsanity.com/paper/2510.16295/) (2025)
- [Position: Life-Logging Video Streams Make the Privacy-Utility Trade-off Inevitable](https://mlsanity.com/paper/2605.10404/) (2026)
- [The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning](https://mlsanity.com/paper/2511.21799/) (2025)
- [Quantifying Membership Disclosure Risk for Tabular Synthetic Data Using Kernel Density Estimators](https://mlsanity.com/paper/2603.10937/) (2026)
- [Membership Inference Attacks Against Video Large Language Models](https://mlsanity.com/paper/2604.27002/) (2026)
- [Membership Inference Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2605.07088/) (2026)
- [A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation](https://mlsanity.com/paper/2605.16090/) (2026)
- [Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignment](https://mlsanity.com/paper/2605.17341/) (2026)
- [Provable Joint Decontamination for Benchmarking Multiple Large Language Models](https://mlsanity.com/paper/2605.21543/) (2026)
- [Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models](https://mlsanity.com/paper/2606.23030/) (2026)
- [One Framework for All: Cross-Modal Membership Inference for Generative Models](https://mlsanity.com/paper/2607.04339/) (2026)
- [When T2I Synthetic Data Backfires: Amplified Privacy Risks in Real-Synthetic Mix Training](https://mlsanity.com/paper/2607.13541/) (2026)
- [MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoning](https://mlsanity.com/paper/2608.17722/) (2026)
- [DIME: Query-Efficient Framework for Membership Inference on Diffusion Models](https://mlsanity.com/paper/2608.22824/) (2026)
- [COMPASS: Membership Inference Attack Against Vision Large Language Models via Adaptive Noise Calibration](https://mlsanity.com/paper/2610.17332/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs), built on pre-trained vision encoders and large language models (LLMs), have shown exceptional multi-modal understanding and dialog capabilities, positioning them as catalysts for the next technological revolution. However, while most VLM research focuses on enhancing multi-modal interaction, the risks of data misuse and leakage have been largely unexplored. This prompts the need for a comprehensive investigation of such risks in VLMs. In this paper, we conduct the first analysis of misuse and leakage detection in VLMs through the lens of membership inference attack (MIA). In specific, we focus on the instruction tuning data of VLMs, which is more likely to contain sensitive or unauthorized information. To address the limitation of existing MIA methods, we introduce a novel approach that infers membership based on a set of samples and their sensitivity to temperature, a unique parameter in VLMs. Based on this, we propose four membership inference methods, each tailored to different levels of background knowledge, ultimately arriving at the most challenging scenario. Our comprehensive evaluations show that these methods can accurately determine membership status, e.g., achieving an AUC greater than 0.8 targeting a small set consisting of only 5 samples on LLaVA.