mlsanityData to 7 Oct 2026

VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

arXiv 2508.02095 ↗First posted 4 Aug 2025cs.CV cs.AIcited by 43

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision language models (VLMs) have shown remarkable capabilities in integrating linguistic and visual reasoning but remain fundamentally limited in understanding dynamic spatiotemporal interactions. Humans effortlessly track and reason about object movements, rotations, and perspective shifts-abilities essential for robust dynamic real-world understanding yet notably lacking in current VLMs. In this paper, we introduce VLM4D, the first benchmark specifically designed to evaluate the spatiotemporal reasoning capabilities of VLMs. Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs emphasizing translational and rotational motions, perspective awareness, and motion continuity. Through comprehensive evaluations of state-of-the-art open and closed-source VLMs, we identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models. Extensive analysis reveals that VLMs struggle particularly with integrating multiple visual cues and maintaining temporal coherence. We further explore promising directions, such as leveraging 4D feature field reconstruction and targeted spatiotemporal supervised fine-tuning, demonstrating their effectiveness in enhancing spatiotemporal comprehension. Our work aims to encourage deeper exploration into improving VLMs' spatial and temporal grounding, paving the way towards more capable and reliable visual intelligence for dynamic environments.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 80 of its 106 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

80 of 106 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  3. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs11 Apr
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan

Other references · 76

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2024GPT-4o System Cardcited by 4,467
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  13. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  14. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  15. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  16. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  17. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  18. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  19. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  20. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  21. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  22. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  23. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  24. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  25. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  26. 2024Phi-4 Technical Reportcited by 675
  27. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  28. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  29. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  30. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  31. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  32. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  33. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  34. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  35. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  36. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  37. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  38. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 31.

43 papers.

Sort:

By area:

  1. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  2. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  3. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  4. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  5. 2025DSI-Bench: A Benchmark for Dynamic Spatial Intelligencecited by 20
  6. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  7. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  8. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 11
  9. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 10
  10. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 8
  11. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  12. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
31 more
  1. 2025FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videoscited by 5
  2. 2026VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?cited by 4
  3. 2026Make Geometry Matter for Spatial Reasoningcited by 4
  4. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 4
  5. 2026NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulationcited by 4
  6. 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
  7. 2026ReMoT: Reinforcement Learning with Motion Contrast Tripletscited by 3
  8. 2026Weakly Supervised Video Anomaly Detection with Anomaly-Connected Components and Intention Reasoningcited by 3
  9. 2026LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialoguecited by 2
  10. 2026Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMscited by 2
  11. 2026MotionVLA: Injecting Geometric Motion into Vision-Language-Action Modelcited by 2
  12. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 1
  13. 2026Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidencecited by 1
  14. 20264DP-QA: Scalable QA for 4D Perception in Vision Language Modelscited by 1
  15. 2026TEMPO: Learning Temporal Context for Dynamic Robot Manipulationcited by 1
  16. 2025Evaluating point-light biological motion in multimodal large language modelscited by 0
  17. 2025Thinking Ahead: Foresight Intelligence in MLLMs and World Modelcited by 0
  18. 2026YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perceptioncited by 0
  19. 2026SpaceDG: Benchmarking Spatial Intelligence under Visual Degradationcited by 0
  20. 2026GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoningcited by 0
  21. 2026The TIME Machine: On The Power of Motion for Efficient Perceptioncited by 0
  22. 2026TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Drivingcited by 0
  23. 2026Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoningcited by 0
  24. 2026DataClaw0: Agentic Tailoring Multimodal Data from Raw Streamscited by 0
  25. 2026Decoding Children's Gait Behaviorcited by 0
  26. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  27. 2026Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Modelscited by 0
  28. 2026AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Modelscited by 0
  29. 2026From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Modelscited by 0
  30. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answeringcited by 0
  31. 2026ChronoGraph: Functional 4D Scene Graphs with Vision-Language Models for Interaction Understanding and Grounded Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2026ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?
  4. 2026Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video
  5. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation
  6. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
  7. 2025ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models
  8. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead