# MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations

arXiv: https://arxiv.org/abs/2503.15871 · HTML: https://mlsanity.com/paper/2503.15871/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 56 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)

## Built on by (most cited first, 22 of 22)

- [Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark](https://mlsanity.com/paper/2504.14693/) (2025)
- [Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Grounding](https://mlsanity.com/paper/2509.11866/) (2025)
- [SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding](https://mlsanity.com/paper/2512.04643/) (2025)
- [Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering](https://mlsanity.com/paper/2505.12826/) (2025)
- [Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting](https://mlsanity.com/paper/2510.02155/) (2025)
- [Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering](https://mlsanity.com/paper/2601.05159/) (2026)
- [ReMoT: Reinforcement Learning with Motion Contrast Triplets](https://mlsanity.com/paper/2603.00461/) (2026)
- [NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Models](https://mlsanity.com/paper/2511.06475/) (2025)
- [Seeing Beyond the Scene: Analyzing and Mitigating Background Bias in Action Recognition](https://mlsanity.com/paper/2512.17953/) (2025)
- [Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs](https://mlsanity.com/paper/2605.22823/) (2026)
- [Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models](https://mlsanity.com/paper/2601.22574/) (2026)
- [Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models](https://mlsanity.com/paper/2602.00559/) (2026)
- [Distorted or Fabricated? A Survey on Hallucination in Video LLMs](https://mlsanity.com/paper/2604.12944/) (2026)
- [Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models](https://mlsanity.com/paper/2604.17873/) (2026)
- [MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Models](https://mlsanity.com/paper/2607.01117/) (2026)
- [VERHallu: Evaluating and Mitigating Event Relation Hallucination in Video Large Language Models](https://mlsanity.com/paper/2601.10010/) (2026)
- [Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models](https://mlsanity.com/paper/2604.12582/) (2026)
- [Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models](https://mlsanity.com/paper/2605.08974/) (2026)
- [MotionHalluc: Diagnosing Kinematic Hallucinations in Fine-Grained Motion Reasoning](https://mlsanity.com/paper/2606.23061/) (2026)
- [HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization](https://mlsanity.com/paper/2607.17994/) (2026)
- [PRISM: Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning](https://mlsanity.com/paper/2608.30388/) (2026)
- [STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models](https://mlsanity.com/paper/2609.29607/) (2026)

## Abstract (arXiv)

In this work, we tackle action-scene hallucination in Video Large Language Models (Video-LLMs), where models incorrectly predict actions based on the scene context or scenes based on observed actions. We observe that existing Video-LLMs often suffer from action-scene hallucination due to two main factors. First, existing Video-LLMs intermingle spatial and temporal features by applying an attention operation across all tokens. Second, they use the standard Rotary Position Embedding (RoPE), which causes the text tokens to overemphasize certain types of tokens depending on their sequential orders. To address these issues, we introduce MASH-VLM, Mitigating Action-Scene Hallucination in Video-LLMs through disentangled spatial-temporal representations. Our approach includes two key innovations: (1) DST-attention, a novel attention mechanism that disentangles the spatial and temporal tokens within the LLM by using masked attention to restrict direct interactions between the spatial and temporal tokens; (2) Harmonic-RoPE, which extends the dimensionality of the positional IDs, allowing the spatial and temporal tokens to maintain balanced positions relative to the text tokens. To evaluate the action-scene hallucination in Video-LLMs, we introduce the UNSCENE benchmark with 1,320 videos and 4,078 QA pairs. Extensive experiments demonstrate that MASH-VLM achieves state-of-the-art results on the UNSCENE benchmark, as well as on existing video understanding benchmarks.