mlsanityData to 14 Oct 2026

Text2Pos: Text-to-Point-Cloud Cross-Modal Localization

arXiv 2203.15125 ↗First posted 28 Mar 2022cs.CV cs.CL cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Natural language-based communication with mobile devices and home appliances is becoming increasingly popular and has the potential to become natural for communicating with mobile robots in the future. Towards this goal, we investigate cross-modal text-to-point-cloud localization that will allow us to specify, for example, a vehicle pick-up or goods delivery location. In particular, we propose Text2Pos, a cross-modal localization module that learns to align textual descriptions with localization cues in a coarse- to-fine manner. Given a point cloud of the environment, Text2Pos locates a position that is specified via a natural language-based description of the immediate surroundings. To train Text2Pos and study its performance, we construct KITTI360Pose, the first dataset for this task based on the recently introduced KITTI360 dataset. Our experiments show that we can localize 65% of textual queries within 15m distance to query locations for top-10 retrieved locations. This is a starting point that we hope will spark future developments towards language-based navigation.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

41 of 58 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  10. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
29 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  2. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  3. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 941
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  7. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  8. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 559
  9. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  10. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  11. 2016DSAC - Differentiable RANSAC for Camera Localizationcited by 266
  12. 2016Segmentation from Natural Language Expressionscited by 257
  13. 2021Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognitioncited by 238
  14. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  15. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  16. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  17. 2019Understanding the Limitations of CNN-based Absolute Camera Pose Regressioncited by 177
  18. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  19. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  20. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  21. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  22. 2017Learning Semantic Concepts and Order for Image and Sentence Matchingcited by 62
  23. 2018VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometrycited by 61
  24. 2019To Learn or Not to Learn: Visual Localization from Essential Matricescited by 61
  25. 2020Consensus-Aware Visual-Semantic Embedding for Image-Text Matchingcited by 59
  26. 2017Semantic Visual Localizationcited by 54
  27. 2020Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoderscited by 47
  28. 2015Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfercited by 39
  29. 2020Transformer Reasoning Network for Image-Text Matching and Retrievalcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 16; 2026: 12.

40 papers.

Sort:

By area:

  1. 2024Where am I? Cross-View Geo-localization with Natural Language Descriptionscited by 38
  2. 2023Text2Loc: 3D Point Cloud Localization from Natural Languagecited by 35
  3. 2023Text to Point Cloud Localization with Relation-Enhanced Transformercited by 20
  4. 2023CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Datacited by 18
  5. 2024"Where am I?" Scene Retrieval with Languagecited by 15
  6. 2025CrossOver: 3D Scene Cross-Modal Alignmentcited by 12
  7. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  8. 2023WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Languagecited by 8
  9. 2025CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Frameworkcited by 8
  10. 2024MSSPlace: Multi-Sensor Place Recognition with Visual and Text Semanticscited by 7
  11. 2024MambaPlace:Text-to-Point-Cloud Cross-Modal Place Recognition with Attention Mamba Mechanismscited by 7
  12. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
28 more
  1. 2025Bridging Text and Vision: A Multi-View Text-Vision Registration Approach for Cross-Modal Place Recognitioncited by 6
  2. 2025Zero-Shot 4D Lidar Panoptic Segmentationcited by 6
  3. 2025Talk2Event: Grounded Understanding of Dynamic Scenes from Event Camerascited by 6
  4. 2025GOTPR: General Outdoor Text-based Place Recognition Using Scene Graph Retrieval with OpenStreetMapcited by 5
  5. 2024DiaLoc: An Iterative Approach to Embodied Dialog Localizationcited by 4
  6. 2025CityLoc: 6DoF Pose Distributional Localization for Text Descriptions in Large-Scale Scenes with Gaussian Representationcited by 4
  7. 2026VLM-Loc: Localization in Point Cloud Maps via Vision-Language Modelscited by 4
  8. 2024Instance-free Text to Point Cloud Localization with Relative Position Awarenesscited by 3
  9. 2025City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learningcited by 3
  10. 2025Range and Bird's Eye View Fused Cross-Modal Visual Place Recognitioncited by 2
  11. 2025L2RSI: Cross-view LiDAR-based Place Recognition for Large-scale Urban Scenes via Remote Sensing Imagerycited by 2
  12. 2025Vehicle-Scene Interaction: A Text-Driven 3D Lidar Place Recognition Method for Autonomous Drivingcited by 2
  13. 2026LaVPR: Benchmarking Language and Vision for Place Recognitioncited by 2
  14. 2024LidaRefer: Context-aware Outdoor 3D Visual Grounding for Autonomous Drivingcited by 1
  15. 2025TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimationcited by 1
  16. 2025Visual Grounding from Event Camerascited by 1
  17. 2026SpatiaLoc: Leveraging Multi-Level Spatial Enhanced Descriptors for Cross-Modal Localizationcited by 1
  18. 2026Global Cross-Modal Geo-Localization: A Million-Scale Dataset and a Physical Consistency Learning Frameworkcited by 1
  19. 2026Riemannian and Symplectic Geometry for Hierarchical Text-Driven Place Recognitioncited by 1
  20. 2026DialogueVPR: Towards Conversational Visual Place Recognitioncited by 1
  21. 2025Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Languagecited by 0
  22. 2025Text2Graph VPR: A Text-to-Graph Expert System for Explainable Place Recognition in Changing Environmentscited by 0
  23. 2026TOL: Textual Localization with OpenStreetMapcited by 0
  24. 2026MAPS: Multi-Anchor Projection Similarity for Joint Vision-Language Geo-Localizationcited by 0
  25. 2026Computing Monetary Risk Measures in Linear Timecited by 0
  26. 2026Trajectory-aware Cross-view Geo-localization with Sequential Observationscited by 0
  27. 2026CitySTAR: Structured and Topology-Aware Reasoning for Open-Vocabulary Urban 3D Groundingcited by 0
  28. 2026PosEviLoc: Position-Conditioned Spatial Evidence for Language-Based 3D Localizationcited by 0