mlsanityData to 7 Oct 2026

4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

arXiv 2512.17012 ↗First posted 18 Dec 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Despite advances in Multimodal LLMs (MLLMs), their ability to reason over 3D structures and temporal dynamics remains limited, constrained by weak 4D perception and temporal understanding. Existing 3D and 4D Video Question Answering (VQA) benchmarks also emphasize static scenes and lack region-level prompting. We tackle these issues by introducing: (a) 4D-RGPT, a specialized MLLM designed to capture 4D representations from video inputs with enhanced temporal perception; (b) Perceptual 4D Distillation (P4D), a training framework that transfers 4D representations from a frozen expert model into 4D-RGPT for comprehensive 4D perception; and (c) R4D-Bench, a benchmark for depth-aware dynamic scenes with region-level prompting, built via a hybrid automated and human-verified pipeline. Our 4D-RGPT achieves notable improvements on both existing 4D VQA benchmarks and the proposed R4D-Bench benchmark.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 90 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

82 of 90 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models9 Oct
  2. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models22 Sep
  3. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model19 Sep
  4. 20253D Aware Region Prompted Vision Language Model16 Sep
  5. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models4 Aug
  6. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs26 Jun
  7. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing11 Jun
  8. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models3 Jun
  9. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  10. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  11. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence29 May
  12. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
9 more
  1. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  2. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models22 May
  3. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding18 May
  4. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models1 May
  5. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoning2 Apr
  6. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?31 Mar
  7. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
  8. 2025ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models25 Mar
  9. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec

Other references · 61

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2021Vision Transformers for Dense Predictioncited by 1,789
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  14. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  16. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  17. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  18. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  19. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  20. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  21. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  22. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  23. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  24. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  25. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  26. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  27. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  28. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  29. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  30. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  31. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  32. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  33. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  34. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  35. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  36. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  37. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  38. 2024CoLLaVO: Crayon Large Language and Vision mOdelcited by 26

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HumanMoveVQA: Can Video MLLMs reason about human movement in videos?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding
  2. 2024Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering
  3. 20264DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking
  4. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
  5. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering
  6. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence
  7. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs
  8. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?