mlsanityData to 7 Oct 2026

Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions

arXiv 2510.08581 ↗First posted 19 Sep 2025cs.SD cs.AI eess.AScited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucinations in multimodal models have been extensively studied using benchmarks that probe reliability in image-text query settings. However, the effect of spoken queries on multimodal hallucinations remains largely unexplored, despite the growing role of voice interfaces. In this paper, we introduce a systematic pipeline that converts existing multimodal hallucination benchmarks into spoken-query versions while preserving the original tasks and labels. We instantiate this pipeline on RePOPE and release RePOPE-Spk, where all queries are provided as spoken audio under diverse input conditions. Experimental results show that hallucinations escalate when queries are spoken rather than written: error rates increase by 3-6% with clean speech and by up to 30% under environmental noise. Furthermore, many-shot prompting and chain-of-thought reasoning provide only partial mitigation. Our findings motivate new directions for building reliable voice interface systems and evaluations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 34 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

14 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding27 May
  2. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio16 Oct
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov

Other references · 11

  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2025Why Language Models Hallucinatecited by 205
  4. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  6. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  7. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  8. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  9. 2025RePOPE: Impact of Annotation Errors on the POPE Benchmarkcited by 9
  10. 2025SpokenNativQA: Multilingual Everyday Spoken Queries for LLMscited by 8
  11. 2025CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models
  2. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
  3. 2026HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
  4. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
  5. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
  6. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
  7. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Model
  8. 2024Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective