# SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning

arXiv: https://arxiv.org/abs/2610.14439 · HTML: https://mlsanity.com/paper/2610.14439/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [TAO: A Large-Scale Benchmark for Tracking Any Object](https://mlsanity.com/paper/2005.10356/) (2020)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses](https://mlsanity.com/paper/1612.00197/) (2016)
- [Referring Multi-Object Tracking](https://mlsanity.com/paper/2303.03366/) (2023)
- [MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2307.15700/) (2023)
- [Multiple Object Tracking as ID Prediction](https://mlsanity.com/paper/2403.16848/) (2024)
- [Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?](https://mlsanity.com/paper/2210.07681/) (2022)
- [UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning](https://mlsanity.com/paper/2509.18094/) (2025)
- [Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking](https://mlsanity.com/paper/2410.01806/) (2024)
- [Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level](https://mlsanity.com/paper/2411.09921/) (2024)
- [HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation](https://mlsanity.com/paper/2411.18042/) (2024)
- [Fostering Video Reasoning via Next-Event Prediction](https://mlsanity.com/paper/2505.22457/) (2025)
- [Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation](https://mlsanity.com/paper/2212.01568/) (2022)
- [Beyond MOT: Semantic Multi-Object Tracking](https://mlsanity.com/paper/2403.05021/) (2024)
- [Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World](https://mlsanity.com/paper/2603.12746/) (2026)
- [FIction: 4D Future Interaction Prediction from Video](https://mlsanity.com/paper/2412.00932/) (2024)
- [OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking](https://mlsanity.com/paper/2410.17534/) (2024)
- [ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2508.07089/) (2025)
- [Motion-o: Trajectory-Grounded Video Reasoning](https://mlsanity.com/paper/2603.18856/) (2026)
- [DynTrace: Tracking Dynamic Object Evidence for 4D Spatio-Temporal Reasoning in MLLMs](https://mlsanity.com/paper/2607.12503/) (2026)
- [Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs](https://mlsanity.com/paper/2607.02425/) (2026)

## Abstract (arXiv)

Large multimodal models (LMMs) need to understand how entities move and interact as a scene develops. This requires following the same entities over time and relating their observed motion to subsequent changes. We formulate Scene Evolution Understanding (SEU), a task for predicting and interpreting this change. Given observed video, a model estimates entity histories and predicts one scene continuation through future trajectories and Events, accompanied by Visual Evidence from observed motion and a scene description. We construct SceneForesee, a benchmark for evaluating entity continuity and scene evolution reasoning through paired observed and future scenes with shared entity identities. It contains 144.57K temporal pairs, 1.05M entity correspondences, and 272.49K Events paired with observed evidence. We propose a spatiotemporal reasoning framework, EvoSee, that learns future geometry and Event semantics through shared candidate states and joint assignment. It selects one continuation per entity from observation and uses the selected changes for entity refinement, retention, and scene representation. On SceneForesee, EvoSee outperforms Qwen3.5-9B by 10.45 points in Combined Point-F1 and 17.24 points in Event Global F1. On the 4D reasoning benchmark Dyn-Bench, it improves question answering accuracy and grounding J&F by 4.8 and 3.58 points over their respective base systems. These results support scene evolution as both a prediction target and a dynamic representation that brings predicted entity changes into spatiotemporal reasoning.The project is available at https://github.com/mikubaka88/SceneForesee-Spatiotemporal-Reasoning.