mlsanityData to 7 Oct 2026

Cowpox: Towards the Immunity of VLM-based Multi-Agent Systems

arXiv 2508.09230 ↗First posted 12 Aug 2025cs.MA cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision Language Model (VLM)-based agents are stateful, autonomous entities capable of perceiving and interacting with their environments through vision and language. Multi-agent systems comprise specialized agents who collaborate to solve a (complex) task. A core security property is robustness, stating that the system should maintain its integrity under adversarial attacks. However, the design of existing multi-agent systems lacks the robustness consideration, as a successful exploit against one agent can spread and infect other agents to undermine the entire system's assurance. To address this, we propose a new defense approach, Cowpox, to provably enhance the robustness of multi-agent systems. It incorporates a distributed mechanism, which improves the recovery rate of agents by limiting the expected number of infections to other agents. The core idea is to generate and distribute a special cure sample that immunizes an agent against the attack before exposure and helps recover the already infected agents. We demonstrate the effectiveness of Cowpox empirically and provide theoretical robustness guarantees.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 36 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

33 of 36 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models26 Jun
  2. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  4. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  6. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  8. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  9. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2016Learning without Forgettingcited by 2,465
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  11. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
12 more
  1. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  2. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  3. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 228
  4. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  5. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  6. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  7. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  8. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  9. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  10. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  11. 2024AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent Systemcited by 53
  12. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2025ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks
  2. 2026REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs
  3. 2025Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models
  4. 2026Adversarial attacks against Modern Vision-Language Models
  5. 2026Agent-Native Immune System: Architecture, Taxonomy, and Engineering
  6. 2025Adversarial Attacks on Robotic Vision Language Action Models
  7. 2025When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs
  8. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments