mlsanityData to 7 Oct 2026

Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

arXiv 2608.20756 ↗First posted 21 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

While multimodal retrieval-augmented generation (RAG) systems increasingly rely on images as external knowledge sources, the introduction of poisoned visual evidence can severely compromise multimodal large language model (MLLM) generation. Unlike prior attacks that rely on altering textual metadata, we introduce Vis-Poison, a novel visual knowledge poisoning attack where the poisoned image itself is the attacker-controlled payload, without manipulating captions, summaries, metadata, or other associated text. Specifically, this attack is instantiated through an automated multi-agent method that constructs visually plausible poisoned images. To assess its impact, we evaluate Vis-Poison across two representative multimodal RAG pipelines, four embedding models, and six generation models. Empirically, Vis-Poison achieves an end-to-end attack success rate of 40.16% to 65.40% against 30k-entry multimodal knowledge bases in black-box settings. Moreover, Vis-Poison remains effective against various MLLMs that can answer correctly from parametric knowledge alone, with an average success rate above 60%. Code and data are available at https://github.com/SWUFE-DB-Group/Vis-Poison.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 33 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

13 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Adversarial attacks against Modern Vision-Language Models17 Mar
  2. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks25 Feb

Other references · 11

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  4. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  5. 2023Using an LLM to Help With Code Understandingcited by 222
  6. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  7. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applicationscited by 31
  8. 2024Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?cited by 17
  9. 2025Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLMcited by 10
  10. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  11. 2022Guiding Visual Question Answering with Attention Priorscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation
  2. 2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
  3. 2026ViTeGate: Visual-Textual Triggered Knowledge Poisoning for Vision-Language Retrieval-Augmented Generation
  4. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation
  5. 2026Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation
  6. 2025Practical Poisoning Attacks against Retrieval-Augmented Generation
  7. 2025POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models
  8. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generation