mlsanityData to 14 Oct 2026

Learning Point-Language Hierarchical Alignment for 3D Visual Grounding

arXiv 2210.12513 ↗First posted 22 Oct 2022cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel hierarchical alignment model (HAM) that learns multi-granularity visual and linguistic representations in an end-to-end manner. We extract key points and proposal points to model 3D contexts and instances, and propose point-language alignment with context modulation (PLACM) mechanism, which learns to gradually align word-level and sentence-level linguistic embeddings with visual representations, while the modulation with the visual context captures latent informative relationships. To further capture both global and local relationships, we propose a spatially multi-granular modeling scheme that applies PLACM to both global and local fields. Experimental results demonstrate the superiority of HAM, with visualized results showing that it can dynamically model fine-grained visual and linguistic representations. HAM outperforms existing methods by a significant margin and achieves state-of-the-art performance on two publicly available datasets, and won the championship in ECCV 2022 ScanRefer challenge. Code is available at~urlhttps://github.com/PPjmchen/HAM.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 61 of its 80 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

61 of 80 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  4. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2016Modeling Context in Referring Expressionscited by 1,063
  7. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  9. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  10. 2018Panoptic Segmentationcited by 802
  11. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  13. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  14. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  15. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  16. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  17. 2016Visual Dialogcited by 377
  18. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  19. 2017Embodied Question Answeringcited by 336
  20. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 326
  21. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  22. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  23. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  24. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  25. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  26. 2021Group-Free 3D Object Detection via Transformerscited by 215
  27. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  28. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  29. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  30. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  31. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138
  32. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  33. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  34. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  35. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  36. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  37. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97
  38. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 1.

6 papers.

  1. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  2. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  3. 2024Transcrib3D: 3D Referring Expression Resolution through Large Language Modelscited by 11
  4. 2024A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenescited by 10
  5. 2025Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understandingcited by 8
  6. 2024R2G: Reasoning to Ground in 3D Scenescited by 2