# CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data

arXiv: https://arxiv.org/abs/2310.18773 · HTML: https://mlsanity.com/paper/2310.18773/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 30 of 45)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark](https://mlsanity.com/paper/1704.03847/) (2017)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [Multi-View Transformer for 3D Visual Grounding](https://mlsanity.com/paper/2204.02174/) (2022)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [SAT: 2D Semantics Assisted Training for 3D Visual Grounding](https://mlsanity.com/paper/2105.11450/) (2021)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2209.14941/) (2022)
- [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021)
- [TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding](https://mlsanity.com/paper/2108.02388/) (2021)

## Built on by (most cited first, 18 of 18)

- [Openfly: A comprehensive platform for aerial vision-language navigation](https://mlsanity.com/paper/2502.18041/) (2025)
- [CityNav: A Large-Scale Dataset for Real-World Aerial Navigation](https://mlsanity.com/paper/2406.14240/) (2024)
- [PointArena: Probing Multimodal Grounding Through Language-Guided Pointing](https://mlsanity.com/paper/2505.09990/) (2025)
- [Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Construction](https://mlsanity.com/paper/2503.11091/) (2025)
- [3EED: Ground Everything Everywhere in 3D](https://mlsanity.com/paper/2511.01755/) (2025)
- [SpatialLLM: From Multi-modality Data to Urban Spatial Intelligence](https://mlsanity.com/paper/2505.12703/) (2025)
- [ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding](https://mlsanity.com/paper/2501.01366/) (2025)
- [AirNav: A Large-Scale UAV Vision-and-Language Navigation Dataset with Natural and Diverse Instructions](https://mlsanity.com/paper/2601.03707/) (2026)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models](https://mlsanity.com/paper/2603.09826/) (2026)
- [ViSA-Enhanced Aerial VLN: A Visual-Spatial Reasoning Enhanced Framework for Aerial Vision-Language Navigation](https://mlsanity.com/paper/2603.08007/) (2026)
- [HTNav: A Hybrid Navigation Framework with Tiered Structure for Urban Aerial Vision-and-Language Navigation](https://mlsanity.com/paper/2604.08883/) (2026)
- [Visual Grounding from Event Cameras](https://mlsanity.com/paper/2509.09584/) (2025)
- [3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding](https://mlsanity.com/paper/2603.23447/) (2026)
- [Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolation](https://mlsanity.com/paper/2604.07923/) (2026)
- [Structured World-State Reasoning for Agentic Robotic Search](https://mlsanity.com/paper/2609.23841/) (2026)
- [Token-Budgeted Candidate Retrieval for 3D Scene-Grounded VLMs: Category Names vs. Pairwise Geometry](https://mlsanity.com/paper/2610.14207/) (2026)
- [Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes](https://mlsanity.com/paper/2610.18356/) (2026)

## Abstract (arXiv)

City-scale 3D point cloud is a promising way to express detailed and complicated outdoor structures. It encompasses both the appearance and geometry features of segmented city components, including cars, streets, and buildings, that can be utilized for attractive applications such as user-interactive navigation of autonomous vehicles and drones. However, compared to the extensive text annotations available for images and indoor scenes, the scarcity of text annotations for outdoor scenes poses a significant challenge for achieving these applications. To tackle this problem, we introduce the CityRefer dataset for city-level visual grounding. The dataset consists of 35k natural language descriptions of 3D objects appearing in SensatUrban city scenes and 5k landmarks labels synchronizing with OpenStreetMap. To ensure the quality and accuracy of the dataset, all descriptions and labels in the CityRefer dataset are manually verified. We also have developed a baseline system that can learn encoded language descriptions, 3D object instances, and geographical information about the city's landmarks to perform visual grounding on the CityRefer dataset. To the best of our knowledge, the CityRefer dataset is the largest city-level visual grounding dataset for localizing specific 3D objects.