mlsanityData to 14 Oct 2026

Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs

arXiv 2411.19187 ↗First posted 28 Nov 2024cs.CLcited by 22

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

The rapid development of Large Multimodal Models (LMMs) has significantly advanced multimodal understanding by harnessing the language abilities of Large Language Models (LLMs) and integrating modality-specific encoders. However, LMMs are plagued by hallucinations that limit their reliability and adoption. While traditional methods to detect and mitigate these hallucinations often involve costly training or rely heavily on external models, recent approaches utilizing internal model features present a promising alternative. In this paper, we critically assess the limitations of the state-of-the-art training-free technique, the logit lens, in handling generalized visual hallucinations. We introduce ContextualLens, a refined method that leverages contextual token embeddings from middle layers of LMMs. This approach significantly improves hallucination detection and grounding across diverse categories, including actions and OCR, while also excelling in tasks requiring contextual understanding, such as spatial relations and attribute comparison. Our novel grounding technique yields highly precise bounding boxes, facilitating a transition from Zero-Shot Object Segmentation to Grounded Visual Question Answering. Our contributions pave the way for more reliable and interpretable multimodal models.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 42 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

24 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations3 Oct
  2. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov

Other references · 21

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  7. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  9. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  10. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  11. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  12. 2023Fine-tuning Language Models for Factualitycited by 221
9 more
  1. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  3. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  4. 2023Retrieving Supporting Evidence for Generative Question Answeringcited by 31
  5. 2024Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answeringcited by 25
  6. 2021Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsulescited by 22
  7. 2021A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanationscited by 14
  8. 2024Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decompositioncited by 12
  9. 2023Sentence Attention Blocks for Answer Groundingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 15.

22 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  3. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  4. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  5. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  6. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  7. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
  8. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Modelscited by 4
  9. 2025Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documentscited by 3
  10. 2026Event-Grounded Sparse Autoencoders for Vision-Language-Action Policiescited by 2
  11. 2026Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attentioncited by 2
  12. 2025Decomposition-Enhanced Training for Post-Hoc Attributions In Language Modelscited by 1
10 more
  1. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  2. 2026Sanity Checks for Long-Form Hallucination Detectioncited by 1
  3. 2026Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Modelscited by 1
  4. 2025RADAR: A Reasoning-Guided Attribution Framework for Explainable Visual Data Analysiscited by 0
  5. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  6. 2026MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answeringcited by 0
  7. 2026TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMscited by 0
  8. 2026PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Modelscited by 0
  9. 2026VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMscited by 0
  10. 2026VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models
  3. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
  4. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding
  5. 2026ROT: Rotating Hidden States towards Contextual Vectors for Hallucination Mitigation in LVLMs
  6. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
  7. 2026UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
  8. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs