mlsanityData to 14 Oct 2026

FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failures

arXiv 2603.06600 ↗First posted 17 Feb 2026cs.LG cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) are prone to errors, and identifying where these errors occur is critical for ensuring the reliability and safety of AI systems. In this paper, we propose an approach that automatically generates questions designed to deliberately induce incorrect responses from VLMs, thereby revealing their vulnerabilities. The core of this approach lies in fuzz testing and reinforcement finetuning: we transform a single input query into a large set of diverse variants through vision and language fuzzing. Based on the fuzzing outcomes, the question generator is further instructed by adversarial reinforcement fine-tuning to produce increasingly challenging queries that trigger model failures. With this approach, we can consistently drive down a target VLM's answer accuracy -- for example, the accuracy of Qwen2.5-VL-32B on our generated questions drops from 86.58% to 65.53% in four RL iterations. Moreover, a fuzzing policy trained against a single target VLM transfers to multiple other VLMs, producing challenging queries that degrade their performance as well.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 49 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

36 of 49 references.

References · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
24 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  5. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  6. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  7. 2019Scene Text Visual Question Answeringcited by 331
  8. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  9. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  10. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  11. 2017Learning to Detect Human-Object Interactionscited by 240
  12. 2018TallyQA: Answering Complex Counting Questionscited by 153
  13. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  14. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  15. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  16. 2023NOPE: Novel Object Pose Estimation from a Single Imagecited by 38
  17. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchycited by 26
  18. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 25
  19. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  20. 2019NLVR2 Visual Bias Analysiscited by 19
  21. 2025NegVQA: Can Vision Language Models Understand Negation?cited by 13
  22. 2024Visual Robustness Benchmark for Visual Question Answering (VQA)cited by 12
  23. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  24. 2023HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025R1-Fuzz: Specializing Language Models for Textual Fuzzing via Reinforcement Learning
  2. 2026Discovering Failure Modes in Vision-Language Models using RL
  3. 2026Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning
  4. 2026Seeing and Solving Are Not Enough for Vision-Language Models
  5. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
  6. 2025VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making
  7. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  8. 2026How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA