mlsanityData to 7 Oct 2026

Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models

arXiv 2512.03121 ↗First posted 2 Dec 2025cs.CR cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Large Multimodal Language Models (MLLMs) are emerging as one of the foundational tools in an expanding range of applications. Consequently, understanding training-data leakage in these systems is increasingly critical. Log-probability-based membership inference attacks (MIAs) have become a widely adopted approach for assessing data exposure in large language models (LLMs), yet their effect in MLLMs remains unclear. We present the first comprehensive evaluation of extending these text-based MIA methods to multimodal settings. Our experiments under vision-and-text (V+T) and text-only (T-only) conditions across the DeepSeek-VL and InternVL model families show that in in-distribution settings, logit-based MIAs perform comparably across configurations, with a slight V+T advantage. Conversely, in out-of-distribution settings, visual inputs act as regularizers, effectively masking membership signals.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 13 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

11 of 13 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Membership Inference Attacks against Large Vision-Language Models5 Nov

Other references · 10

  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2016A Diagram Is Worth A Dozen Imagescited by 793
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  8. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  9. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  10. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 61

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FiMMIA: scaling semantic perturbation-based membership inference across modalities
  2. 2025Membership Inference Attacks Against Vision-Language Models
  3. 2026Detecting Data Contamination in Large Language Models
  4. 2025OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models
  5. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing
  6. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Models
  7. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  8. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models