mlsanityData to 7 Oct 2026

Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness

arXiv 2504.01901 ↗First posted 2 Apr 2025cs.CV cs.AI cs.CLcited by 61

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The rapid development of Large Multimodal Models (LMMs) for 2D images and videos has spurred efforts to adapt these models for interpreting 3D scenes. However, the absence of large-scale 3D vision-language datasets has posed a significant obstacle. To address this issue, typical approaches focus on injecting 3D awareness into 2D LMMs by designing 3D input-level scene representations. This work provides a new perspective. We introduce reconstructive visual instruction tuning with 3D-awareness (Ross3D), which integrates 3D-aware visual supervision into the training procedure. Specifically, it incorporates cross-view and global-view reconstruction. The former requires reconstructing masked views by aggregating overlapping information from other views. The latter aims to aggregate information from all available views to recover Bird's-Eye-View images, contributing to a comprehensive overview of the entire scene. Empirically, Ross3D achieves state-of-the-art performance across various 3D scene understanding benchmarks. More importantly, our semi-supervised experiments demonstrate significant potential in leveraging large amounts of unlabeled 3D vision-only data.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 98 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

72 of 98 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  3. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  4. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  5. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2024Qwen2.5 Technical Reportcited by 4,713
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  12. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  13. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  14. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  15. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  16. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  17. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  18. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  19. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  20. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  21. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  22. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  23. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  24. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  25. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  26. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  27. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  28. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  29. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  30. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  31. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  32. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  33. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  34. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  35. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  36. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  37. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  38. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 33.

61 papers.

Sort:

By area:

  1. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  2. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  3. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  4. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  5. 2025VGR: Visual Grounded Reasoningcited by 70
  6. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  7. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  8. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  9. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  10. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  11. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 28
  12. 2025Reasoning in Space via Grounding in the Worldcited by 18
38 more
  1. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  2. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  3. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 14
  4. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  5. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 9
  6. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  7. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  8. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialoguescited by 7
  9. 2026Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuningcited by 7
  10. 2025Video Spatial Reasoning with Object-Centric 3D Rolloutcited by 6
  11. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 6
  12. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  13. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidancecited by 5
  14. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  15. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  16. 2026Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulatorscited by 4
  17. 2025POMA-3D: The Point Map Way to 3D Scene Understandingcited by 3
  18. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  19. 2025EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligencecited by 3
  20. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splattingcited by 3
  21. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  22. 2026Z3D: Zero-Shot 3D Visual Grounding from Imagescited by 2
  23. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  24. 2026C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoningcited by 2
  25. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  26. 2026Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignmentcited by 2
  27. 2026Unlocking Dense Metric Depth Estimation in VLMscited by 2
  28. 2025Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understandingcited by 1
  29. 2025SpatialReasoner: Active Perception for Large-Scale 3D Scene Understandingcited by 1
  30. 2026Modeling Cross-vision Synergy for Unified Large Vision Modelcited by 1
  31. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  32. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 1
  33. 2026PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Modelscited by 1
  34. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  35. 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoningcited by 1
  36. 2025Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referringcited by 0
  37. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 0
  38. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reg3D: Reconstructive Geometry Instruction Tuning for 3D Scene Understanding
  2. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
  3. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  4. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning
  5. 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
  6. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  7. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
  8. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understanding