mlsanityData to 14 Oct 2026

OpenCity3D: What do Vision-Language Models know about Urban Environments?

arXiv 2503.16776 ↗First posted 21 Mar 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) show great promise for 3D scene understanding but are mainly applied to indoor spaces or autonomous driving, focusing on low-level tasks like segmentation. This work expands their use to urban-scale environments by leveraging 3D reconstructions from multi-view aerial imagery. We propose OpenCity3D, an approach that addresses high-level tasks, such as population density estimation, building age classification, property price prediction, crime rate assessment, and noise pollution evaluation. Our findings highlight OpenCity3D's impressive zero-shot and few-shot capabilities, showcasing adaptability to new contexts. This research establishes a new paradigm for language-driven urban analytics, enabling applications in planning, policy, and environmental monitoring. See our project page: opencity3d.github.io

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

30 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views4 Apr

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  7. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  8. 2023LERF: Language Embedded Radiance Fieldscited by 543
  9. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  10. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  11. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  12. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
17 more
  1. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  2. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  3. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  4. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  5. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  6. 2022STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Datasetcited by 53
  7. 2023AGILE3D: Attention Guided Interactive Multi-object 3D Segmentationcited by 33
  8. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  9. 2024SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphscited by 29
  10. 20223D Segmentation of Humans in Point Clouds with Synthetic Datacited by 24
  11. 2024ICGNet: A Unified Approach for Instance-Centric Graspingcited by 16
  12. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  13. 2024Spot-Compose: A Framework for Open-Vocabulary Object Retrieval and Drawer Manipulation in Point Cloudscited by 15
  14. 2021Automated 3D reconstruction of LoD2 and LoD1 models for all 10 million buildings of the Netherlandscited by 14
  15. 2023LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectoriescited by 14
  16. 2022ImpliCity: City Modeling from Satellite Images with Deep Implicit Occupancy Fieldscited by 9
  17. 2024ARKit LabelMaker: A New Scale for Indoor 3D Scene Understandingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025SLAG: Scalable Language-Augmented Gaussian Splattingcited by 4
  2. 2025Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agendacited by 2
  3. 2026Can Urban Blight Be Accessed with Vision-language Models: A Case Study in Detroitcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20263DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
  2. 2025GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fields
  3. 2024Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments
  4. 2025How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images
  5. 2025City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
  6. 2024An Introduction to Vision-Language Modeling
  7. 2025PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models
  8. 2025Do Vision-Language Models See Urban Scenes as People Do? An Urban Perception Benchmark