# Unified Hallucination Fuzzing for Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2608.07525 · HTML: https://mlsanity.com/paper/2608.07525/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 30 of 44)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [Detecting and Preventing Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2308.06394/) (2023)
- [Detecting and Evaluating Medical Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2406.10185/) (2024)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset](https://mlsanity.com/paper/2403.11116/) (2024)
- [MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models](https://mlsanity.com/paper/2502.14302/) (2025)
- [MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning](https://mlsanity.com/paper/2505.16964/) (2025)
- [HallE-Control: Controlling Object Hallucination in Large Multimodal Models](https://mlsanity.com/paper/2310.01779/) (2023)
- [AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models](https://mlsanity.com/paper/2406.10900/) (2024)
- [OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation](https://mlsanity.com/paper/2411.18499/) (2024)
- [Mitigating Open-Vocabulary Caption Hallucinations](https://mlsanity.com/paper/2312.03631/) (2023)
- [Hallucination Benchmark in Medical Visual Question Answering](https://mlsanity.com/paper/2401.05827/) (2024)
- [BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models](https://mlsanity.com/paper/2407.13442/) (2024)

## Built on by (most cited first, 1 of 1)

- [Improving Generalization Robustness of Multimodal RLVR](https://mlsanity.com/paper/2608.08802/) (2026)

## Abstract (arXiv)

Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applications. Existing evaluations, predominantly based on static benchmarks, suffer from narrow taxonomical coverage and rapid performance saturation, failing to reflect model robustness in evolving real-world scenarios. To bridge this gap, we present a systematic evaluation framework integrating a comprehensive benchmark with self-evolving stress testing. First, we introduce UniHall, a fine-grained dataset grounded in a unified taxonomy spanning Object, Instruction, and Knowledge dimensions. Second, to address benchmark saturation, we propose Self-Adaptive Multimodal Fuzzing (SAMF), a self-adaptive framework that employs evolutionary mutation strategies to explore the boundaries of model hallucinations. Crucially, to ensure reliable assessment of dynamic inputs, SAMF incorporates a structured metric suite driven by an ensemble of multi-modal oracles. Our extensive experiments reveal that state-of-the-art MLLMs exhibit significant performance degradation under fuzzing compared to conventional settings, exposing a dissociation between reasoning capabilities and factual grounding. Furthermore, we identify a helpfulness-hallucination trade-off, where reinforcement learning alignment inadvertently exacerbates sycophancy in instruction-following tasks. The framework, code and benchmark are available at https://github.com/LanceZPF/EvalHall.