mlsanityData to 7 Oct 2026

Understanding Multimodal Hallucination with Parameter-Free Representation Alignment

arXiv 2409.01151 ↗First posted 2 Sep 2024cs.CV cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucination is a common issue in Multimodal Large Language Models (MLLMs), yet the underlying principles remain poorly understood. In this paper, we investigate which components of MLLMs contribute to object hallucinations. To analyze image representations while completely avoiding the influence of all other factors other than the image representation itself, we propose a parametric-free representation alignment metric (Pfram) that can measure the similarities between any two representation systems without requiring additional training parameters. Notably, Pfram can also assess the alignment of a neural representation system with the human representation system, represented by ground-truth annotations of images. By evaluating the alignment with object annotations, we demonstrate that this metric shows strong and consistent correlations with object hallucination across a wide range of state-of-the-art MLLMs, spanning various model architectures and sizes. Furthermore, using this metric, we explore other key issues related to image representations in MLLMs, such as the role of different modules, the impact of textual instructions, and potential improvements including the use of alternative visual encoders. Our code is available at: https://github.com/yellow-binary-tree/Pfram.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 66 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

49 of 66 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models24 Jun
  2. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  3. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  4. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  8. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  9. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  10. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  11. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  12. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
3 more
  1. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  2. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
22 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2020Understanding the Behaviour of Contrastive Losscited by 397
  10. 2024The Platonic Representation Hypothesiscited by 362
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  13. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  14. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  15. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  16. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  17. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  18. 2021Generalized Shape Metrics on Neural Representationscited by 109
  19. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25
  20. 2022Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learningcited by 20
  21. 2024Visually Dehallucinative Instruction Generation: Know What You Don't Knowcited by 10
  22. 2020Inter-layer Information Similarity Assessment of Deep Neural Networks Via Topological Similarity and Persistence Analysis of Data Neighbour Dynamicscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  2. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  2. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  3. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  4. 2026Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM
  5. 2026What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustness
  6. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
  7. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  8. 2026Fine-Grained Multi Image Object Hallucination Benchmark