mlsanityData to 7 Oct 2026

OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models

arXiv 2609.11244 ↗First posted 10 Sep 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress across diverse tasks, they suffer from hallucinations where generated outputs contradict or misrepresent input semantics. Existing research typically addresses hallucination detection within a single modality or task type, limiting generalizability. We introduce OmniHallu, a unified hallucination detection framework spanning both comprehension and generation tasks across image, video, and audio modalities. We contribute OmniHallu-Bench, a 10,000-sample benchmark with claim-level human annotations covering six cross-modal tasks: image-to-text (I2T), video-to-text (V2T), audio-to-text (A2T), text-to-image (T2I), text-to-video (T2V), and text-to-audio (T2A). Our multi-agent architecture decomposes model outputs into atomic claims, verifies them through modality-specific experts, and aggregates evidence via structured reasoning. We further propose a preference-optimized trainable verifier that approximates the multi-agent decision boundary, reducing expert calls by 66% with minimal performance loss. Extensive experiments reveal a consistent modality-dependent performance gradient and provide fine-grained insights into cross-modal hallucination patterns.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 55 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

33 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models24 Jun
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov

Other references · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  8. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  12. 2019Clotho: An Audio Captioning Datasetcited by 354
19 more
  1. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  2. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  3. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  4. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  5. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  6. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  7. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  8. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  9. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  10. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  11. 2023Models See Hallucinations: Evaluating the Factuality in Video Captioningcited by 19
  12. 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Modelscited by 18
  13. 2024Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Modelscited by 17
  14. 2024On the Audio Hallucinations in Large Audio-Video Language Modelscited by 15
  15. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  16. 2021The MSR-Video to Text Dataset with Clean Annotationscited by 8
  17. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  18. 2025MIST: Towards Multi-dimensional Implicit BiaS Evaluation of LLMs for Theory of Mindcited by 1
  19. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detection
  2. 2024Unified Hallucination Detection for Multimodal Large Language Models
  3. 2026ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation
  4. 2026HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
  5. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
  6. 2026Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models
  7. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  8. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs