mlsanityData to 7 Oct 2026

Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models

arXiv 2605.27101 ↗First posted 26 May 2026cs.CV cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A key capability for video understanding is reliably linking subjects to events across time, yet whether Video Large Language Models (VideoLLMs) actually achieve this remains unclear. In this work, we introduce DistractionBench to evaluate whether VideoLLMs can robustly link subjects and events in the presence of unrelated video segments. Through controlled interventions, such as inserting short advertisement clips into longer videos, we show that VideoLLMs frequently hallucinate interactions between entities from different segments, incorrectly attributing actions from injected advertisements to subjects in the main video. We characterize this systematic hallucination as bag-of-events (BoE) behavior, where models process videos as collections of events rather than temporally structured sequences. Evaluating 11 popular VideoLLMs, we find that all models exhibit substantial BoE behavior. Our findings suggest that VideoLLMs lack reliable mechanisms for temporal grounding and motivate the development of models with more robust subject-event association.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

15 of 23 references.

References · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  7. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  8. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  9. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  10. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  11. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  12. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
3 more
  1. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  2. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  3. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32

Built on by

Not cited yet.