mlsanityData to 14 Oct 2026

Unifying 3D Vision-Language Understanding via Promptable Queries

arXiv 2405.11442 ↗First posted 19 May 2024cs.CVcited by 70

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A unified model for 3D vision-language (3D-VL) understanding is expected to take various scene representations and perform a wide range of tasks in a 3D scene. However, a considerable gap exists between existing methods and such a unified model, due to the independent application of representation and insufficient exploration of 3D multi-task training. In this paper, we introduce PQ3D, a unified model capable of using Promptable Queries to tackle a wide range of 3D-VL tasks, from low-level instance segmentation to high-level reasoning and planning. This is achieved through three key innovations: (1) unifying various 3D scene representations (i.e., voxels, point clouds, multi-view images) into a shared 3D coordinate space by segment-level grouping, (2) an attention-based query decoder for task-specific information retrieval guided by prompts, and (3) universal output heads for different tasks to support multi-task training. Tested across ten diverse 3D-VL datasets, PQ3D demonstrates impressive performance on these tasks, setting new records on most benchmarks. Particularly, PQ3D improves the state-of-the-art on ScanNet200 by 4.9% (AP25), ScanRefer by 5.4% ([email protected]), Multi3DRefer by 11.7% ([email protected]), and Scan2Cap by 13.4% ([email protected]). Moreover, PQ3D supports flexible inference with individual or combined forms of available 3D representations, e.g., solely voxel input.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 70 of its 81 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

70 of 81 references.

References · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Segment Anythingcited by 8,234
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
38 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  4. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  5. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  7. 2023LERF: Language Embedded Radiance Fieldscited by 543
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  11. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  13. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  14. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  15. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  16. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  17. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  18. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  19. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  20. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  21. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  22. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  23. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  24. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  25. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  26. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  27. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  28. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  29. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  30. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  31. 2021Instances as Queriescited by 158
  32. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  33. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  34. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  35. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  36. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  37. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  38. 2022SimpleClick: Interactive Image Segmentation with Simple Vision Transformerscited by 111

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 38; 2026: 21.

70 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  3. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  4. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  5. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  6. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  7. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  8. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  9. 2025Detect Anything 3D in the Wildcited by 33
  10. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  11. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  12. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
38 more
  1. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  2. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  3. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  4. 2025Reasoning in Space via Grounding in the Worldcited by 19
  5. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  6. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  7. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  8. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  9. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  10. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  11. 2025DSPNet: Dual-vision Scene Perception for Robust 3D Question Answeringcited by 9
  12. 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysiscited by 8
  13. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8
  14. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 8
  15. 2024CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splattingcited by 7
  16. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  17. 2025From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMscited by 7
  18. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  19. 2025SAB3R: Semantic-Augmented Backbone in 3D Reconstructioncited by 6
  20. 2025Cook and Clean Together: Teaching Embodied Agents for Parallel Task Executioncited by 6
  21. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  22. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  23. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understandingcited by 5
  24. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  25. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  26. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  27. 2025DEGround: An Effective Baseline for Ego-centric 3D Visual Grounding with a Homogeneous Frameworkcited by 4
  28. 2025GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Groundingcited by 4
  29. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Modelscited by 3
  30. 2025Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptionscited by 3
  31. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  32. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  33. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  34. 2025Zero-Shot 3D Visual Grounding from Vision-Language Modelscited by 2
  35. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRIcited by 2
  36. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  37. 2026Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignmentcited by 2
  38. 2026Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMscited by 2

50 most cited of 70 All 70 (JSON)