# VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

arXiv: https://arxiv.org/abs/2508.02095 · HTML: https://mlsanity.com/paper/2508.02095/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 80 of its 106 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 80)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)

## Built on by (most cited first, 30 of 46)

- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [SpatialVID: A Large-Scale Video Dataset with Spatial Annotations](https://mlsanity.com/paper/2509.09676/) (2025)
- [SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition](https://mlsanity.com/paper/2511.21471/) (2025)
- [Holistic Evaluation of Multimodal LLMs on Spatial Intelligence](https://mlsanity.com/paper/2508.13142/) (2025)
- [DSI-Bench: A Benchmark for Dynamic Spatial Intelligence](https://mlsanity.com/paper/2510.18873/) (2025)
- [SpaceVista: All-Scale Visual Spatial Reasoning from mm to km](https://mlsanity.com/paper/2510.09606/) (2025)
- [Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?](https://mlsanity.com/paper/2602.07055/) (2026)
- [I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models](https://mlsanity.com/paper/2509.16072/) (2025)
- [Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2603.13615/) (2026)
- [MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence](https://mlsanity.com/paper/2603.00515/) (2026)
- [Make Geometry Matter for Spatial Reasoning](https://mlsanity.com/paper/2603.26639/) (2026)
- [CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning](https://mlsanity.com/paper/2601.13304/) (2026)
- [Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents](https://mlsanity.com/paper/2605.08747/) (2026)
- [NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2607.06678/) (2026)
- [FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos](https://mlsanity.com/paper/2512.10927/) (2025)
- [SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments](https://mlsanity.com/paper/2604.14144/) (2026)
- [VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?](https://mlsanity.com/paper/2603.07888/) (2026)
- [Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmark](https://mlsanity.com/paper/2601.00092/) (2025)
- [ReMoT: Reinforcement Learning with Motion Contrast Triplets](https://mlsanity.com/paper/2603.00461/) (2026)
- [Weakly Supervised Video Anomaly Detection with Anomaly-Connected Components and Intention Reasoning](https://mlsanity.com/paper/2603.00550/) (2026)
- [4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation](https://mlsanity.com/paper/2512.17012/) (2025)
- [LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue](https://mlsanity.com/paper/2605.19390/) (2026)
- [Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs](https://mlsanity.com/paper/2605.22823/) (2026)
- [MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model](https://mlsanity.com/paper/2606.08288/) (2026)
- [Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence](https://mlsanity.com/paper/2603.13091/) (2026)
- [SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation](https://mlsanity.com/paper/2605.22536/) (2026)
- [4DP-QA: Scalable QA for 4D Perception in Vision Language Models](https://mlsanity.com/paper/2606.11568/) (2026)
- [Decoding Children's Gait Behavior](https://mlsanity.com/paper/2608.00371/) (2026)
- [TEMPO: Learning Temporal Context for Dynamic Robot Manipulation](https://mlsanity.com/paper/2609.16864/) (2026)
- [Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering](https://mlsanity.com/paper/2609.38177/) (2026)

## Abstract (arXiv)

Vision language models (VLMs) have shown remarkable capabilities in integrating linguistic and visual reasoning but remain fundamentally limited in understanding dynamic spatiotemporal interactions. Humans effortlessly track and reason about object movements, rotations, and perspective shifts-abilities essential for robust dynamic real-world understanding yet notably lacking in current VLMs. In this paper, we introduce VLM4D, the first benchmark specifically designed to evaluate the spatiotemporal reasoning capabilities of VLMs. Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs emphasizing translational and rotational motions, perspective awareness, and motion continuity. Through comprehensive evaluations of state-of-the-art open and closed-source VLMs, we identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models. Extensive analysis reveals that VLMs struggle particularly with integrating multiple visual cues and maintaining temporal coherence. We further explore promising directions, such as leveraging 4D feature field reconstruction and targeted spatiotemporal supervised fine-tuning, demonstrating their effectiveness in enhancing spatiotemporal comprehension. Our work aims to encourage deeper exploration into improving VLMs' spatial and temporal grounding, paving the way towards more capable and reliable visual intelligence for dynamic environments.