mlsanityData to 14 Oct 2026

LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue

arXiv 2605.19390 ↗First posted 19 May 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent large multimodal models (LMMs) have become increasingly capable on image and video understanding, yet still struggle to sustain 4D continuous spatiotemporal dynamic reasoning. To study this capability gap, we formulate trajectory-grounded multi-turn spatiotemporal dialogue, a new task in which a model must answer spatiotemporal queries while returning structured 3D target trajectories over an entire short clip or a specified segment of a longer clip, and introduce Track4D-Bench, a benchmark with 526 clip-level dialogue samples spanning 23.5k frames and 7.5k object annotations, for training and evaluation. Building on this task, we propose LMM-Track4D, which combines RTGE (Ray--Time Geometry Encoding), a dedicated streaming state token TRK for long-horizon dynamic propagation, and an Object-Slot Kinematic, Residual-Anchor (OSK-RA) decoder for stable 4-step 3D state estimation under occlusion and viewpoint variation. Experiments on Track4D-Bench show consistent improvements over strong baselines, suggesting that explicit dynamic state modeling is a useful design principle for eliciting 4D dynamic reasoning in LMMs. Our code and dataset will be publicly available at https://github.com/mikubaka88/LMM-Track4D.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 51 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

44 of 51 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World13 Mar
  2. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation28 Sep
  3. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models4 Aug
  4. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding31 Jul
  5. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models3 Jun
  6. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  7. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
  8. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding18 May
  9. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
  10. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  11. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  12. 2024Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering2 Jun
1 more
  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan

Other references · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  10. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  11. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
  12. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
19 more
  1. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  2. 2023DynamicStereo: Consistent Dynamic Depth from Stereo Videoscited by 148
  3. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 127
  4. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
  5. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  6. 2025AI for Service: Proactive Assistance with AI Glassescited by 20
  7. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  8. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  9. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agentscited by 9
  10. 2026Innovator-VL: A Multimodal Large Language Model for Scientific Discoverycited by 8
  11. 2025VISTA: Mitigating Semantic Inertia in Video-LLMs via Training-Free Dynamic Chain-of-Thought Routingcited by 7
  12. 2026DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentcited by 7
  13. 2025Understanding Dynamic Scenes in Ego Centric 4D Point Cloudscited by 6
  14. 2025ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splattingcited by 6
  15. 2026Flash-Unified: A Training-Free and Task-Aware Acceleration Framework for Native Unified Modelscited by 5
  16. 2026EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistantcited by 5
  17. 20263D Instruction Ambiguity Detectioncited by 3
  18. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
  19. 2025Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understandingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splattingcited by 3
  2. 2026Dyn-3D: Unveiling and Resolving Ego-Motion Ambiguity in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20264DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking
  2. 2026Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM
  3. 2026DynTrace: Tracking Dynamic Object Evidence for 4D Spatio-Temporal Reasoning in MLLMs
  4. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation
  5. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models
  6. 2025Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models
  7. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos
  8. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding