# SPIKE-RL: Video-LLMs meet Bayesian Surprise

arXiv: https://arxiv.org/abs/2509.23433 · HTML: https://mlsanity.com/paper/2509.23433/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 27 of 27)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [Adaptive Keyframe Sampling for Long Video Understanding](https://mlsanity.com/paper/2502.21271/) (2025)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [M-LLM Based Video Frame Selection for Efficient Video Understanding](https://mlsanity.com/paper/2502.19680/) (2025)
- [Frame-Voyager: Learning to Query Frames for Video Large Language Models](https://mlsanity.com/paper/2410.03226/) (2024)
- [Oops! Predicting Unintentional Action in Video](https://mlsanity.com/paper/1911.11206/) (2019)
- [Koala: Key frame-conditioned long video-LLM](https://mlsanity.com/paper/2404.04346/) (2024)
- [KeyVideoLLM: Towards Large-scale Video Keyframe Selection](https://mlsanity.com/paper/2407.03104/) (2024)
- [FunQA: Towards Surprising Video Comprehension](https://mlsanity.com/paper/2306.14899/) (2023)
- [Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding](https://mlsanity.com/paper/2503.13139/) (2025)
- [Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Models](https://mlsanity.com/paper/2502.11881/) (2025)
- [Enough Coin Flips Can Make LLMs Act Bayesian](https://mlsanity.com/paper/2503.04722/) (2025)
- [HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding](https://mlsanity.com/paper/2503.08585/) (2025)
- [Video Summarization with Large Language Models](https://mlsanity.com/paper/2504.11199/) (2025)
- [Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering](https://mlsanity.com/paper/2401.10711/) (2024)
- [Can Language Models Laugh at YouTube Short-form Videos?](https://mlsanity.com/paper/2310.14159/) (2023)
- [Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events](https://mlsanity.com/paper/2412.05725/) (2024)
- [Leveraging Self-Supervised Training for Unintentional Action Recognition](https://mlsanity.com/paper/2209.11870/) (2022)

## Built on by (most cited first, 1 of 1)

- [YoCausal: How Far is Video Generation from World Model? A Causality Perspective](https://mlsanity.com/paper/2605.30346/) (2026)

## Abstract (arXiv)

Real-world videos often show routine activities punctuated by memorable, surprising events. However, most Video-LLMs process videos by sampling frames uniformly, likely missing critical moments that define a video's narrative. We introduce SPIKE, an inference-time framework that quantifies Bayesian Surprise as the belief update triggered by new visual evidence in the video stream, identifying moments where new visual evidence conflicts with prior beliefs. SPIKE effectively localizes surprise in videos, strongly correlated with humans on positive (FunQA) and negative (Oops!) surprise benchmarks. Since the beliefs of zero-shot Video-LLMs are often suboptimal, we develop SPIKE-RL, which leverages GRPO to optimize belief hypotheses based on a reward signal from the video caption. SPIKE and SPIKE-RL guide query-agnostic surprise-weighted frame sampling, which allocates more frames to interesting moments in the video. With this strategy, we achieve consistent performance gains on five downstream benchmarks over uniform sampling. By enabling Video-LLMs to track beliefs and register surprise, our work paves the way for more robust models that can revise their understanding in response to new information.