mlsanityData to 7 Oct 2026

Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation

arXiv 2603.00172 ↗First posted 26 Feb 2026cs.CR cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for enhancing multimodal large language models by grounding their responses in external, factual knowledge and thus mitigating hallucinations. However, the integration of externally sourced knowledge bases introduces a critical attack surface. Adversaries can inject malicious multimodal content capable of influencing both retrieval and downstream generation. In this work, we present MM-MEPA, a multimodal poisoning attack that targets the metadata components of image-text entries while leaving the associated visual content unaltered. By only manipulating the metadata, MM-MEPA can still steer multimodal retrieval and induce attacker-desired model responses. We evaluate the attack across multiple benchmark settings and demonstrate its severity. MM-MEPA achieves an attack success rate of up to 91% consistently disrupting system behaviors across four retrievers and two multimodal generators. Additionally, we assess representative defense strategies and find them largely ineffective against this form of metadata-only poisoning. Our findings expose a critical vulnerability in multimodal RAG and underscore the urgent need for more robust, defense-aware retrieval and knowledge integration methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 59 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

46 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks25 Feb

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
33 more
  1. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  6. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  7. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  8. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  9. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  10. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  11. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  12. 2021WebQA: Multihop and Multimodal QAcited by 109
  13. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  14. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  15. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  16. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  17. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbationscited by 64
  18. 2024"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Modelscited by 60
  19. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 52
  20. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  21. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  22. 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
  23. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  24. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  25. 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 35
  26. 2025Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Modelscited by 32
  27. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  28. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  29. 2023Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairscited by 15
  30. 2025MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answeringcited by 9
  31. 2024IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listingcited by 5
  32. 2025Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generationcited by 2
  33. 2025PoultryTalk: A Multi-modal Retrieval-Augmented Generation (RAG) System for Intelligent Poultry Management and Decision Supportcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation
  2. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation
  3. 2026Walking the Embedding Space: Datastore Extraction from Multimodal RAG
  4. 2026Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems
  5. 2025RAGPart & RAGMask: Retrieval-Stage Defenses Against Corpus Poisoning in Retrieval-Augmented Generation
  6. 2024Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models
  7. 2025Uncovering Competing Poisoning Attacks in Retrieval-Augmented Generation
  8. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation