mlsanityData to 13 Oct 2026

FINER: MLLMs Hallucinate under Fine-grained Negative Queries

arXiv 2603.17662 ↗First posted 18 Mar 2026cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FIne-grained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and ``what'' questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at hrefhttps://explainableml.github.io/finer-project/https://explainableml.github.io/finer-project/.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 62 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

48 of 62 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Robust Multimodal Large Language Models Against Modality Conflict9 Jul
  2. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling17 Apr
  3. 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMs30 Mar
  4. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  5. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs28 Jan
  6. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
  7. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models14 Sep
  8. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs2 Aug
  9. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning22 Jul
  10. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  11. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
5 more
  1. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  4. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  3. 2025Qwen3 Technical Reportcited by 8,510
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  5. 2025Qwen2.5-VL Technical Reportcited by 5,499
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,714
  8. 2019Towards VQA Models That Can Readcited by 1,705
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
  10. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,452
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,450
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
19 more
  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 899
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 723
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 708
  4. 2024Phi-4 Technical Reportcited by 696
  5. 2018Object Hallucination in Image Captioningcited by 663
  6. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 470
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 446
  8. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 374
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 323
  10. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  11. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 131
  12. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 96
  13. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  14. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 87
  15. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  16. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 30
  17. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  18. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  19. 2025RePOPE: Impact of Annotation Errors on the POPE Benchmarkcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluationcited by 1
  2. 2026Reliability-Prioritized Fine-Grained Generation in Multimodal Largecited by 0
  3. 2026NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
  2. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
  3. 2025Measuring Epistemic Humility in Multimodal Large Language Models
  4. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models
  5. 2024LargePiG: Your Large Language Model is Secretly a Pointer Generator
  6. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  7. 2025HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models
  8. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models