mlsanityData to 7 Oct 2026

The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineages

arXiv 2606.15821 ↗First posted 14 Jun 2026cs.CL cs.AI cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advances in large language models (LLMs) have produced many specialized multimodal LLMs (MLLMs) that share common foundational LLMs, forming distinct model lineages. It remains unclear whether a fundamental behavioral link exists between the foundational LLMs and downstream variants. We investigate this question by quantifying head-level context-truthfulness scores. Across diverse LLM and MLLM lineages, including Vicuna-, Qwen2.5-, LLaMA2-, and Mistral-based models, we find that Truth Scores are strongly preserved within model families, even after instruction tuning or multimodal adaptation. We further show that this inheritance is consistent with attention-head weight preservation, and that context-truthful heads attend to query-relevant evidence. Building on this finding, we propose TruthProbe, a soft-gating strategy that amplifies context-truthful heads while preserving other head contributions. TruthProbe improves contextual truthfulness on HaluEval and reduces multimodal hallucination on POPE and CHAIR, with base-LLM Truth Scores transferring effectively to their fine-tuned LLM and MLLM descendants. Code is available at https://github.com/miso-choi/TruthProbe.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 59 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

54 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs24 Feb
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  4. 2024The Revolution of Multimodal Large Language Models: A Survey19 Feb
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  7. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 47

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
35 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2023Vision Transformers Need Registerscited by 844
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2024Large Language Models: A Surveycited by 506
  7. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  10. 2024InternLM2 Technical Reportcited by 391
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  12. 2023How Language Model Hallucinations Can Snowballcited by 310
  13. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  14. 2025Why Language Models Hallucinatecited by 205
  15. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  16. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  17. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  18. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  19. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  20. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  21. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  22. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  23. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  24. 2024Attention Heads of Large Language Models: A Surveycited by 65
  25. 2023Improving Visual Prompt Tuning for Self-supervised Vision Transformerscited by 57
  26. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  27. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  28. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignmentcited by 42
  29. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  30. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 26
  31. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  32. 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformerscited by 16
  33. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 14
  34. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  35. 2026Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformerscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different setcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating LLM Hallucinations through Domain-Grounded Tiered Retrieval
  2. 2025OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models
  3. 2026A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs
  4. 2025Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
  5. 2025HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models
  6. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
  7. 2026Anchored Confabulation: Partial Evidence Non-Monotonically Amplifies Confident Hallucination in LLMs
  8. 2026Grounding Without Corrective Control: Truth-Tracking Profiles for Large Language Models