# SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapse

arXiv: https://arxiv.org/abs/2512.18671 · HTML: https://mlsanity.com/paper/2512.18671/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 21 Dec 2025.

## Builds on (most cited first, 18 of 18)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos](https://mlsanity.com/paper/2501.13826/) (2025)
- [Revisiting Catastrophic Forgetting in Large Language Model Tuning](https://mlsanity.com/paper/2406.04836/) (2024)
- [Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models](https://mlsanity.com/paper/2408.02032/) (2024)
- [VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding](https://mlsanity.com/paper/2412.03735/) (2024)
- [TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning](https://mlsanity.com/paper/2504.09641/) (2025)
- [Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.10150/) (2024)
- [PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning](https://mlsanity.com/paper/2504.05810/) (2025)
- [ARGUS: Hallucination and Omission Evaluation in Video-LLMs](https://mlsanity.com/paper/2506.07371/) (2025)
- [TimeSoccer: An End-to-End Multimodal Large Language Model for Soccer Commentary Generation](https://mlsanity.com/paper/2504.17365/) (2025)

## Built on by (most cited first, 9 of 9)

- [TIR-Flow: Active Video Search and Reasoning with Frozen VLMs](https://mlsanity.com/paper/2601.06176/) (2026)
- [Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models](https://mlsanity.com/paper/2601.22574/) (2026)
- [STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Models](https://mlsanity.com/paper/2604.03045/) (2026)
- [Distorted or Fabricated? A Survey on Hallucination in Video LLMs](https://mlsanity.com/paper/2604.12944/) (2026)
- [When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models](https://mlsanity.com/paper/2604.17375/) (2026)
- [Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models](https://mlsanity.com/paper/2605.08974/) (2026)
- [MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models](https://mlsanity.com/paper/2606.11792/) (2026)
- [It Lied to a Doctor to Buy Poison Ingredients: Quantifying Real-World Misuse of Phone-use Agents](https://mlsanity.com/paper/2606.27944/) (2026)
- [STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models](https://mlsanity.com/paper/2609.29607/) (2026)

## Abstract (arXiv)

Despite Video Large Language Models having rapidly advanced in recent years, perceptual hallucinations pose a substantial safety risk, which severely restricts their real-world applicability. While several methods for hallucination mitigation have been proposed, they often compromise the model's capacity for video understanding and reasoning. In this work, we propose SmartSight, a pioneering step to address this issue in a training-free manner by leveraging the model's own introspective capabilities. Specifically, SmartSight generates multiple candidate responses to uncover low-hallucinated outputs that are often obscured by standard greedy decoding. It assesses the hallucination of each response using the Temporal Attention Collapse score, which measures whether the model over-focuses on trivial temporal regions of the input video when generating the response. To improve efficiency, SmartSight identifies the Visual Attention Vanishing point, enabling more accurate hallucination estimation and early termination of hallucinated responses, leading to a substantial reduction in decoding cost. Experiments show that SmartSight substantially lowers hallucinations for Qwen2.5-VL-7B by 10.59% on VRIPT-HAL, while simultaneously enhancing video understanding and reasoning, boosting performance on VideoMMMU by up to 8.86%. These results highlight SmartSight's effectiveness in improving the reliability of open-source Video-LLMs.