Two Sides of the Same Coin: Co-Evolving Search for Cross-Task Attacks on Vision-Language Models
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
Abstract (arXiv)
Vision-language models (VLMs) exhibit strong generalization across multimodal tasks but remain vulnerable to adversarial perturbations. Existing attacks typically follow single-trajectory gradient optimization or task-specific objectives, limiting search-space exploration and cross-task transferability. We propose an evolutionary-computation-guided cross-modal attack framework for unified VLMs. The framework adaptively searches both textual and visual spaces. On the textual side, it evolves hard negative semantic embeddings around the source-category representation to provide diverse cross-modal repulsion. On the visual side, it maintains a population of object-region perturbations and combines momentum-based gradient updates with evolutionary selection, mutation, and crossover to more reliably explore multiple feasible trajectories. Jointly optimizing semantic negative guidance and localized perturbations generates adversarial examples that consistently shift source-object semantics toward target categories across vision-language tasks. Theoretical analyses show that the co-evolutionary search preserves perturbation feasibility, prevents degradation of the best observed fitness, and increases the probability of reaching high-margin adversarial regions compared with single-trajectory optimization. Experiments on Florence-2, OFA, and UnifiedIO-2 demonstrate strong overall attack performance across image captioning, object detection, region categorization, and object localization. Ablation studies further verify the complementary effectiveness of text-side semantic evolution and image-side perturbation evolution, as well as the framework's efficiency and cross-task transferability.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 61 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
10 of 61 references.
Closest ancestors · 4
Same direction, newest first
- 2025Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models17 Mar
- 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
- 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
- 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
Other references · 6
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2013Intriguing properties of neural networkscited by 5,114
- 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
- 2024TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Abilitycited by 2
- 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMs
- 2026Breaking the weakest link to evade vision language models
- 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
- 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
- 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
- 2025When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models
- 2025One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models
- 2026Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability