mlsanityData to 14 Oct 2026

QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models

arXiv 2504.11038 ↗First posted 15 Apr 2025cs.CV cs.AIcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

In typical multimodal tasks, such as Visual Question Answering (VQA), adversarial attacks targeting a specific image and question can lead large vision-language models (LVLMs) to provide incorrect answers. However, it is common for a single image to be associated with multiple questions, and LVLMs may still answer other questions correctly even for an adversarial image attacked by a specific question. To address this, we introduce the query-agnostic visual attack (QAVA), which aims to create robust adversarial examples that generate incorrect responses to unspecified and unknown questions. Compared to traditional adversarial attacks focused on specific images and questions, QAVA significantly enhances the effectiveness and efficiency of attacks on images when the question is unknown, achieving performance comparable to attacks on known target questions. Our research broadens the scope of visual adversarial attacks on LVLMs in practical settings, uncovering previously overlooked vulnerabilities, particularly in the context of visual adversarial threats. The code is available at https://github.com/btzyd/qava.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 41 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

29 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  2. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  3. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  4. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 25

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  12. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
13 more
  1. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  3. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  5. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  7. 2018nocaps: novel object captioning at scalecited by 444
  8. 2018Generating Natural Language Adversarial Examplescited by 424
  9. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  10. 2022Frequency-driven Imperceptible Adversarial Attack on Semantic Similaritycited by 77
  11. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  12. 2016Generating Natural Questions About an Imagecited by 56
  13. 2023Enhancing Generalization of Universal Adversarial Perturbation through Gradient Aggregationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 8
  2. 2025VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Modelscited by 4
  3. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  4. 2025Superplatforms Have to Attack AI Agentscited by 3
  5. 2025Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMscited by 1
  6. 2026From Clouds to Hallucinations: Atmospheric Retrieval Hijacking in Remote Sensing Vision-Language RAGcited by 0
  7. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks
  2. 2025Adversarial Robustness of Vision in Open Foundation Models
  3. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models
  4. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  5. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  6. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions
  7. 2025VisionTrap: Unanswerable Questions On Visual Data
  8. 2025Coordinated Robustness Evaluation Framework for Vision-Language Models