# Text to Point Cloud Localization with Relation-Enhanced Transformer

arXiv: https://arxiv.org/abs/2301.05372 · HTML: https://mlsanity.com/paper/2301.05372/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 13 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021)
- [To Learn or Not to Learn: Visual Localization from Essential Matrices](https://mlsanity.com/paper/1908.01293/) (2019)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Text2Pos: Text-to-Point-Cloud Cross-Modal Localization](https://mlsanity.com/paper/2203.15125/) (2022)

## Built on by (most cited first, 20 of 20)

- [Text2Loc: 3D Point Cloud Localization from Natural Language](https://mlsanity.com/paper/2311.15977/) (2023)
- ["Where am I?" Scene Retrieval with Language](https://mlsanity.com/paper/2404.14565/) (2024)
- [UniLoc: Towards Universal Place Recognition Using Any Single Modality](https://mlsanity.com/paper/2412.12079/) (2024)
- [CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Framework](https://mlsanity.com/paper/2503.02593/) (2025)
- [MSSPlace: Multi-Sensor Place Recognition with Visual and Text Semantics](https://mlsanity.com/paper/2407.15663/) (2024)
- [MambaPlace:Text-to-Point-Cloud Cross-Modal Place Recognition with Attention Mamba Mechanisms](https://mlsanity.com/paper/2408.15740/) (2024)
- [Bridging Text and Vision: A Multi-View Text-Vision Registration Approach for Cross-Modal Place Recognition](https://mlsanity.com/paper/2502.14195/) (2025)
- [GOTPR: General Outdoor Text-based Place Recognition Using Scene Graph Retrieval with OpenStreetMap](https://mlsanity.com/paper/2501.08575/) (2025)
- [CityLoc: 6DoF Pose Distributional Localization for Text Descriptions in Large-Scale Scenes with Gaussian Representation](https://mlsanity.com/paper/2501.08982/) (2025)
- [VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models](https://mlsanity.com/paper/2603.09826/) (2026)
- [Instance-free Text to Point Cloud Localization with Relative Position Awareness](https://mlsanity.com/paper/2404.17845/) (2024)
- [Vehicle-Scene Interaction: A Text-Driven 3D Lidar Place Recognition Method for Autonomous Driving](https://mlsanity.com/paper/2503.18035/) (2025)
- [Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models](https://mlsanity.com/paper/2603.18002/) (2026)
- [PointCloud-Text Matching: Benchmark Datasets and a Baseline](https://mlsanity.com/paper/2403.19386/) (2024)
- [TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimation](https://mlsanity.com/paper/2508.08038/) (2025)
- [SpatiaLoc: Leveraging Multi-Level Spatial Enhanced Descriptors for Cross-Modal Localization](https://mlsanity.com/paper/2601.03579/) (2026)
- [Riemannian and Symplectic Geometry for Hierarchical Text-Driven Place Recognition](https://mlsanity.com/paper/2604.01598/) (2026)
- [Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language](https://mlsanity.com/paper/2511.15308/) (2025)
- [TOL: Textual Localization with OpenStreetMap](https://mlsanity.com/paper/2604.01644/) (2026)
- [PosEviLoc: Position-Conditioned Spatial Evidence for Language-Based 3D Localization](https://mlsanity.com/paper/2609.23534/) (2026)

## Abstract (arXiv)

Automatically localizing a position based on a few natural language instructions is essential for future robots to communicate and collaborate with humans. To approach this goal, we focus on the text-to-point-cloud cross-modal localization problem. Given a textual query, it aims to identify the described location from city-scale point clouds. The task involves two challenges. 1) In city-scale point clouds, similar ambient instances may exist in several locations. Searching each location in a huge point cloud with only instances as guidance may lead to less discriminative signals and incorrect results. 2) In textual descriptions, the hints are provided separately. In this case, the relations among those hints are not explicitly described, leading to difficulties of learning relations. To overcome these two challenges, we propose a unified Relation-Enhanced Transformer (RET) to improve representation discriminability for both point cloud and natural language queries. The core of the proposed RET is a novel Relation-enhanced Self-Attention (RSA) mechanism, which explicitly encodes instance (hint)-wise relations for the two modalities. Moreover, we propose a fine-grained cross-modal matching method to further refine the location predictions in a subsequent instance-hint matching stage. Experimental results on the KITTI360Pose dataset demonstrate that our approach surpasses the previous state-of-the-art method by large margin.