# Text2Pos: Text-to-Point-Cloud Cross-Modal Localization

arXiv: https://arxiv.org/abs/2203.15125 · HTML: https://mlsanity.com/paper/2203.15125/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings](https://mlsanity.com/paper/1803.08495/) (2018)

## Built on by (most cited first, 30 of 40)

- [Where am I? Cross-View Geo-localization with Natural Language Descriptions](https://mlsanity.com/paper/2412.17007/) (2024)
- [Text2Loc: 3D Point Cloud Localization from Natural Language](https://mlsanity.com/paper/2311.15977/) (2023)
- [Text to Point Cloud Localization with Relation-Enhanced Transformer](https://mlsanity.com/paper/2301.05372/) (2023)
- [CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data](https://mlsanity.com/paper/2310.18773/) (2023)
- ["Where am I?" Scene Retrieval with Language](https://mlsanity.com/paper/2404.14565/) (2024)
- [CrossOver: 3D Scene Cross-Modal Alignment](https://mlsanity.com/paper/2502.15011/) (2025)
- [3EED: Ground Everything Everywhere in 3D](https://mlsanity.com/paper/2511.01755/) (2025)
- [WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language](https://mlsanity.com/paper/2304.05645/) (2023)
- [CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Framework](https://mlsanity.com/paper/2503.02593/) (2025)
- [MSSPlace: Multi-Sensor Place Recognition with Visual and Text Semantics](https://mlsanity.com/paper/2407.15663/) (2024)
- [MambaPlace:Text-to-Point-Cloud Cross-Modal Place Recognition with Attention Mamba Mechanisms](https://mlsanity.com/paper/2408.15740/) (2024)
- [HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models](https://mlsanity.com/paper/2409.10419/) (2024)
- [Bridging Text and Vision: A Multi-View Text-Vision Registration Approach for Cross-Modal Place Recognition](https://mlsanity.com/paper/2502.14195/) (2025)
- [Zero-Shot 4D Lidar Panoptic Segmentation](https://mlsanity.com/paper/2504.00848/) (2025)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [GOTPR: General Outdoor Text-based Place Recognition Using Scene Graph Retrieval with OpenStreetMap](https://mlsanity.com/paper/2501.08575/) (2025)
- [DiaLoc: An Iterative Approach to Embodied Dialog Localization](https://mlsanity.com/paper/2403.06846/) (2024)
- [CityLoc: 6DoF Pose Distributional Localization for Text Descriptions in Large-Scale Scenes with Gaussian Representation](https://mlsanity.com/paper/2501.08982/) (2025)
- [VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models](https://mlsanity.com/paper/2603.09826/) (2026)
- [Instance-free Text to Point Cloud Localization with Relative Position Awareness](https://mlsanity.com/paper/2404.17845/) (2024)
- [City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning](https://mlsanity.com/paper/2507.12795/) (2025)
- [Range and Bird's Eye View Fused Cross-Modal Visual Place Recognition](https://mlsanity.com/paper/2502.11742/) (2025)
- [L2RSI: Cross-view LiDAR-based Place Recognition for Large-scale Urban Scenes via Remote Sensing Imagery](https://mlsanity.com/paper/2503.11245/) (2025)
- [Vehicle-Scene Interaction: A Text-Driven 3D Lidar Place Recognition Method for Autonomous Driving](https://mlsanity.com/paper/2503.18035/) (2025)
- [LaVPR: Benchmarking Language and Vision for Place Recognition](https://mlsanity.com/paper/2602.03253/) (2026)
- [LidaRefer: Context-aware Outdoor 3D Visual Grounding for Autonomous Driving](https://mlsanity.com/paper/2411.04351/) (2024)
- [TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimation](https://mlsanity.com/paper/2508.08038/) (2025)
- [Visual Grounding from Event Cameras](https://mlsanity.com/paper/2509.09584/) (2025)
- [SpatiaLoc: Leveraging Multi-Level Spatial Enhanced Descriptors for Cross-Modal Localization](https://mlsanity.com/paper/2601.03579/) (2026)
- [Global Cross-Modal Geo-Localization: A Million-Scale Dataset and a Physical Consistency Learning Framework](https://mlsanity.com/paper/2603.08491/) (2026)

## Abstract (arXiv)

Natural language-based communication with mobile devices and home appliances is becoming increasingly popular and has the potential to become natural for communicating with mobile robots in the future. Towards this goal, we investigate cross-modal text-to-point-cloud localization that will allow us to specify, for example, a vehicle pick-up or goods delivery location. In particular, we propose Text2Pos, a cross-modal localization module that learns to align textual descriptions with localization cues in a coarse- to-fine manner. Given a point cloud of the environment, Text2Pos locates a position that is specified via a natural language-based description of the immediate surroundings. To train Text2Pos and study its performance, we construct KITTI360Pose, the first dataset for this task based on the recently introduced KITTI360 dataset. Our experiments show that we can localize 65% of textual queries within 15m distance to query locations for top-10 retrieved locations. This is a starting point that we hope will spark future developments towards language-based navigation.