mlsanityData to 14 Oct 2026

ALOHa: A New Measure for Hallucination in Captioning Models

arXiv 2404.02904 ↗First posted 3 Apr 2024cs.CV cs.AI cs.CLcited by 45

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite recent advances in multimodal pre-training for visual description, state-of-the-art models still produce captions containing errors, such as hallucinating objects not present in a scene. The existing prominent metric for object hallucination, CHAIR, is limited to a fixed set of MS COCO objects and synonyms. In this work, we propose a modernized open-vocabulary metric, ALOHa, which leverages large language models (LLMs) to measure object hallucinations. Specifically, we use an LLM to extract groundable objects from a candidate caption, measure their semantic similarity to reference objects from captions and object detections, and use Hungarian matching to produce a final hallucination score. We show that ALOHa correctly identifies 13.6% more hallucinated objects than CHAIR on HAT, a new gold-standard subset of MS COCO Captions annotated for hallucinations, and 30.8% more on nocaps, where objects extend beyond MS COCO categories. Our code is available at https://davidmchan.github.io/aloha/.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

28 of 47 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
16 more
  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  2. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  5. 2021MetaICL: Learning to Learn In Contextcited by 454
  6. 2018nocaps: novel object captioning at scalecited by 444
  7. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  8. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  9. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  10. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  11. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  12. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  13. 2023IC3: Image Captioning by Committee Consensuscited by 20
  14. 2022Improved Beam Search for Hallucination Mitigation in Abstractive Summarizationcited by 10
  15. 2022Evaluating and Improving Factuality in Multimodal Abstractive Summarizationcited by 6
  16. 2022ED-FAITH: Evaluating Dialogue Summarization on Faithfulnesscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 17; 2026: 20.

45 papers.

Sort:

By area:

  1. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 90
  2. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  3. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 30
  4. 2025Enough Coin Flips Can Make LLMs Act Bayesiancited by 22
  5. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
  6. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 18
  7. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Modelscited by 16
  8. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  9. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  10. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  11. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  12. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
33 more
  1. 2025EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Modelscited by 7
  2. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  3. 2024Extract Free Dense Misalignment from CLIPcited by 5
  4. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Modelscited by 5
  5. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  6. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 5
  7. 2024Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Datacited by 4
  8. 2025VIBE: Can a VLM Read the Room?cited by 4
  9. 2026Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarkingcited by 4
  10. 2025OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Modelscited by 2
  11. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  12. 2026When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluationcited by 2
  13. 2026DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captionscited by 2
  14. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  15. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  16. 2025Metropolis-Hastings Captioning Game: Knowledge Fusion of Vision Language Models via Decentralized Bayesian Inferencecited by 1
  17. 2025The Devil is in the EOS: Sequence Training for Detailed Image Captioningcited by 1
  18. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  19. 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applicationscited by 1
  20. 2025HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioningcited by 0
  21. 2025A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimatescited by 0
  22. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0
  23. 2026SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characterscited by 0
  24. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Modelcited by 0
  25. 2026Symbal: Detecting Systematic Misalignments in Model-Generated Captionscited by 0
  26. 2026Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioningcited by 0
  27. 2026CAPEval: A Decoupled Caption Evaluation across Understanding and Generationcited by 0
  28. 2026Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Modelscited by 0
  29. 2026VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMscited by 0
  30. 2026Can We Trust Video Hallucination Detectors? VidHalLoc for Evaluating the Evaluatorscited by 0
  31. 2026Spot-the-shift: Evaluating Grounded Image Difference Captioning of Long-term Changescited by 0
  32. 2026A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributionscited by 0
  33. 2026Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioningcited by 0