mlsanityData to 7 Oct 2026

Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment

arXiv 2605.08064 ↗First posted 8 May 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Spatial intelligence in vision-language models (VLMs) attracts research interest with the practical demand to reason in the 3D world.Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D geometric priors lack efficiency in vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality. Given only video frames as input, we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation alignment, we further curate the SpaceSpan dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 57 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

53 of 57 references.

Closest ancestors · 14

Same direction, newest first

  1. 20253D Aware Region Prompted Vision Language Model16 Sep
  2. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligence18 Aug
  3. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  4. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
  5. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  6. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  7. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoning2 Apr
  8. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  9. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  10. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  11. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model1 Aug
  12. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
2 more
  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 39

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
27 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  3. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  6. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  8. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  9. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  10. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  11. 2016Segmentation from Natural Language Expressionscited by 257
  12. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  13. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  14. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  15. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  16. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  17. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  18. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  19. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 92
  20. 2023FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformercited by 72
  21. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  22. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  23. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  24. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  25. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  26. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  27. 2025Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMscited by 0
  2. 2026From Alignment to Fusion in 3D Vision-Languagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
  2. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  3. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
  4. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  5. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
  6. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
  7. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  8. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models