mlsanityData to 7 Oct 2026

FiMMIA: scaling semantic perturbation-based membership inference across modalities

arXiv 2512.02786 ↗First posted 2 Dec 2025cs.LGcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Membership Inference Attacks (MIAs) aim to determine whether a specific data point was included in the training set of a target model. Although there are have been numerous methods developed for detecting data contamination in large language models (LLMs), their performance on multimodal LLMs (MLLMs) falls short due to the instabilities introduced through multimodal component adaptation and possible distribution shifts across multiple inputs. In this work, we investigate multimodal membership inference and address two issues: first, by identifying distribution shifts in the existing datasets, and second, by releasing an extended baseline pipeline to detect them. We also generalize the perturbation-based membership inference methods to MLLMs and release FiMMIA -- a modular Framework for Multimodal MIA.footnoteThe source code and framework have been made publicly available under the MIT license via hrefhttps://github.com/ai-forever/dataleakagedetectlink.The video demonstration is available on hrefhttps://youtu.be/a9L4-H80aSgYouTube. Our approach trains a neural network to analyze the target model's behavior on perturbed inputs, capturing distributional differences between members and non-members. Comprehensive evaluations on various fine-tuned multimodal models demonstrate the effectiveness of our perturbation-based membership inference attacks in multimodal domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 42 of its 60 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

42 of 60 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing3 Nov
  2. 2025Machine Text Detectors are Membership Inference Attacks22 Oct
  3. 2025The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage13 Aug
  4. 2024Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attacks31 Aug
  5. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Models23 Jun
  6. 2024Semantic Membership Inference Attack against Large Language Models14 Jun
  7. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration10 Nov
  8. 2023Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study29 Sep
  9. 2023Towards More Realistic Membership Inference Attacks on Large Diffusion Models22 Jun
  10. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  11. 2022M^4I: Multi-modal Models Membership Inference15 Sep
  12. 2021Membership Inference Attacks From First Principles7 Dec
1 more
  1. 2021Membership Inference on Word Embedding and Beyond21 Jun

Other references · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
17 more
  1. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  2. 2020Label-Only Membership Inference Attackscited by 371
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  4. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  5. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  6. 2020Information Leakage in Embedding Modelscited by 199
  7. 2019Reconstruction and Membership Inference Attacks against Generative Modelscited by 122
  8. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 83
  9. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  10. 2025How much do language models memorize?cited by 53
  11. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 38
  12. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  13. 2022Explicit Tradeoffs between Adversarial and Natural Distributional Robustnesscited by 20
  14. 2024Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detectioncited by 14
  15. 2024MIA-Tuner: Adapting Large Language Models as Pre-training Text Detectorcited by 13
  16. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMcited by 9
  17. 2025Multimodal Evaluation of Russian-language Architecturescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models
  2. 2026Detecting Data Contamination in Large Language Models
  3. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  4. 2025Membership Inference Attacks Against Vision-Language Models
  5. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models
  6. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Models
  7. 2024Membership Inference Attacks against Large Vision-Language Models
  8. 2025OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models