# Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage

arXiv: https://arxiv.org/abs/2412.15484 · HTML: https://mlsanity.com/paper/2412.15484/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 48 of its 60 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)

## Built on by (most cited first, 20 of 20)

- [Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.01419/) (2025)
- [From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model](https://mlsanity.com/paper/2510.19871/) (2025)
- [CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning](https://mlsanity.com/paper/2602.21655/) (2026)
- [CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning](https://mlsanity.com/paper/2606.09393/) (2026)
- [On the Generalization Capacities of MLLMs for Spatial Intelligence](https://mlsanity.com/paper/2603.06704/) (2026)
- [Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization](https://mlsanity.com/paper/2505.15291/) (2025)
- [Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs](https://mlsanity.com/paper/2506.03195/) (2025)
- [AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioning](https://mlsanity.com/paper/2507.12841/) (2025)
- [CapGeo-Bench: Decoupling Visual Perception from Reasoning and Evaluating Geometric Understanding](https://mlsanity.com/paper/2510.09302/) (2025)
- [How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People](https://mlsanity.com/paper/2602.13469/) (2026)
- [ReflectCAP: Detailed Image Captioning with Reflective Memory](https://mlsanity.com/paper/2604.12357/) (2026)
- [VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning](https://mlsanity.com/paper/2605.28023/) (2026)
- [Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation](https://mlsanity.com/paper/2606.07647/) (2026)
- [Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact](https://mlsanity.com/paper/2506.15480/) (2025)
- [AGIC: Attention-Guided Image Captioning to Improve Caption Relevance](https://mlsanity.com/paper/2508.06853/) (2025)
- [Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning](https://mlsanity.com/paper/2603.01696/) (2026)
- [DualFact+: A Multimodal Fact Verification Framework for Procedural Video Understanding](https://mlsanity.com/paper/2604.25584/) (2026)
- [Towards Faithful Sentimental Image Captioning via Evidence-Aware Multi-Agent Reasoning](https://mlsanity.com/paper/2607.25789/) (2026)
- [CAPEval: A Decoupled Caption Evaluation across Understanding and Generation](https://mlsanity.com/paper/2608.02589/) (2026)
- [A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss](https://mlsanity.com/paper/2609.00591/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MLLM factuality may fall short in hyper-detailed image captioning tasks. In contrast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions. Our code and data are available at https://github.com/adobe-research/CapMAS.