# Fostering Video Reasoning via Next-Event Prediction

arXiv: https://arxiv.org/abs/2505.22457 · HTML: https://mlsanity.com/paper/2505.22457/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 45 of its 65 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models](https://mlsanity.com/paper/2407.12772/) (2024)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)

## Built on by (most cited first, 18 of 18)

- [Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO](https://mlsanity.com/paper/2511.16669/) (2025)
- [FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs](https://mlsanity.com/paper/2601.13836/) (2026)
- [VideoSSR: Video Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2511.06281/) (2025)
- [Video-CoE: Reinforcing Video Event Prediction via Chain of Events](https://mlsanity.com/paper/2603.14935/) (2026)
- [Scaling GUI Agents with Visual State Transitions](https://mlsanity.com/paper/2607.24112/) (2026)
- [When Thinking Hurts: Mitigating Visual Forgetting in Video Reasoning via Frame Repetition](https://mlsanity.com/paper/2603.16256/) (2026)
- [Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction](https://mlsanity.com/paper/2606.05769/) (2026)
- [EgoSelf: From Memory to Personalized Egocentric Assistant](https://mlsanity.com/paper/2604.19564/) (2026)
- [EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction](https://mlsanity.com/paper/2510.21786/) (2025)
- [MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Prediction](https://mlsanity.com/paper/2601.03781/) (2026)
- [EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs](https://mlsanity.com/paper/2604.23348/) (2026)
- [LogiShot: Logically Coherent Cross-Shot Video Generation](https://mlsanity.com/paper/2608.08820/) (2026)
- [Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning](https://mlsanity.com/paper/2608.15869/) (2026)
- [RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models](https://mlsanity.com/paper/2601.03369/) (2026)
- [FeVOS: Foresight Expression Video Object Segmentation](https://mlsanity.com/paper/2606.25585/) (2026)
- [From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2610.04139/) (2026)
- [SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning](https://mlsanity.com/paper/2610.14439/) (2026)
- [EventLens: Event-Structure Reinforcement Learning for Video Understanding](https://mlsanity.com/paper/2610.16172/) (2026)

## Abstract (arXiv)

Next-token prediction serves as the foundational learning task enabling reasoning in LLMs. But what should the learning task be when aiming to equip MLLMs with temporal reasoning capabilities over video inputs? Existing tasks such as video question answering often rely on annotations from humans or much stronger MLLMs, while video captioning tends to entangle temporal reasoning with spatial information. To address this gap, we propose next-event prediction (NEP), a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning. We segment each video into past and future frames: the MLLM takes the past frames as input and predicts a summary of events derived from the future frames, thereby encouraging the model to reason temporally in order to complete the task. To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios. We further explore a range of video instruction-tuning strategies to study their effects on temporal reasoning. To evaluate progress, we introduce FutureBench to assess coherence in predicting unseen future events. Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.