mlsanityData to 7 Oct 2026

ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance

arXiv 2303.16894 ↗First posted 29 Mar 2023cs.CV cs.AI cs.CLcited by 70

Not on the map. Why?

Abstract (arXiv)

Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.5%, and +1.35% on Sr3D, Nr3D, and ScanRefer. Code is released at https://github.com/Ivan-Tang-3D/ViewRefer3D.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 58 of its 67 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

58 of 67 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
38 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  2. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2018Panoptic Segmentationcited by 796
  5. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  6. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  7. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  8. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  12. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  13. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  14. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  15. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  16. 2019M3D-RPN: Monocular 3D Region Proposal Network for Object Detectioncited by 251
  17. 2021Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognitioncited by 236
  18. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  19. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  20. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  21. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  22. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  23. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  24. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  25. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  26. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  27. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  28. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  29. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  30. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  31. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  32. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  33. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  34. 2022Can Language Understand Depth?cited by 66
  35. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  36. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  37. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  38. 2022Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detectioncited by 48

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 26; 2025: 23; 2026: 10.

70 papers.

Sort:

By area:

  1. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  2. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  3. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  4. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  5. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  6. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  7. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  8. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  9. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  10. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  11. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  12. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
38 more
  1. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  2. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  3. 2024MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Groundingcited by 29
  4. 2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detectioncited by 24
  5. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  6. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  7. 20243D Vision and Language Pretraining with Large-Scale Synthetic Datacited by 21
  8. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  9. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 17
  10. 2024"Where am I?" Scene Retrieval with Languagecited by 15
  11. 2025DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Groundingcited by 12
  12. 2023MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognitioncited by 11
  13. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  14. 2023LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognitioncited by 10
  15. 2025AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referringcited by 10
  16. 2024PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Modelcited by 9
  17. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  18. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  19. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8
  20. 20253EED: Ground Everything Everywhere in 3Dcited by 8
  21. 2026MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentationcited by 8
  22. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6
  23. 2024LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Controlcited by 6
  24. 2024Query3D: LLM-Powered Open-Vocabulary Scene Segmentation with Language Embedded 3D Gaussiancited by 5
  25. 20243D Spatial Understanding in MLLMs: Disambiguation and Evaluationcited by 5
  26. 2025ProxyTransformation: Preshaping Point Cloud Manifold With Proxy Attention For 3D Visual Groundingcited by 5
  27. 2025Empowering Large Language Models with 3D Situation Awarenesscited by 5
  28. 2024SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attentioncited by 4
  29. 2024Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attentioncited by 4
  30. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  31. 2025DEGround: An Effective Baseline for Ego-centric 3D Visual Grounding with a Homogeneous Frameworkcited by 4
  32. 2025GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Groundingcited by 4
  33. 2025AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Usecited by 3
  34. 2025POMA-3D: The Point Map Way to 3D Scene Understandingcited by 3
  35. 2024Multi-Task Domain Adaptation for Language Grounding with 3D Objectscited by 2
  36. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  37. 2025Zero-Shot 3D Visual Grounding from Vision-Language Modelscited by 2
  38. 2024Enhancing Zero-shot Counting via Language-guided Exemplar Learningcited by 1

50 most cited of 70 All 70 (JSON)