# 3D Question Answering for City Scene Understanding arXiv: https://arxiv.org/abs/2407.17398 ยท HTML: https://mlsanity.com/paper/2407.17398/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 52 references that are in the map. - First posted on arXiv 24 Jul 2024. ## Builds on (most cited first, 29 of 29) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015) - [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016) - [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019) - [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017) - [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022) - [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023) - [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020) - [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023) - [Building Generalizable Agents with a Realistic and Rich 3D Environment](https://mlsanity.com/paper/1801.02209/) (2018) - [Embodied Question Answering in Photorealistic Environments with Point Cloud Perception](https://mlsanity.com/paper/1904.03461/) (2019) - [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023) - [Multi-Target Embodied Question Answering](https://mlsanity.com/paper/1904.04686/) (2019) - [3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents](https://mlsanity.com/paper/1908.04929/) (2019) - [CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes](https://mlsanity.com/paper/2304.06061/) (2023) - [SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds](https://mlsanity.com/paper/2201.04494/) (2022) - [CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense](https://mlsanity.com/paper/1908.02962/) (2019) - [Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline](https://mlsanity.com/paper/2209.12028/) (2022) - [UrbanBIS: a Large-scale Benchmark for Fine-grained Urban Building Instance Segmentation](https://mlsanity.com/paper/2305.02627/) (2023) - [MOSS: End-to-End Dialog System Framework with Modular Supervision](https://mlsanity.com/paper/1909.05528/) (2019) - [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022) - [Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation](https://mlsanity.com/paper/2112.11691/) (2021) - [Towards Augmented Reality-driven Human-City Interaction: Current Research on Mobile Headsets and Future Challenges](https://mlsanity.com/paper/2007.09207/) (2020) ## Built on by (most cited first, 10 of 10) - [CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space](https://mlsanity.com/paper/2502.12532/) (2025) - [SpatialLLM: From Multi-modality Data to Urban Spatial Intelligence](https://mlsanity.com/paper/2505.12703/) (2025) - [ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding](https://mlsanity.com/paper/2501.01366/) (2025) - [RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System](https://mlsanity.com/paper/2511.18286/) (2025) - [Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding](https://mlsanity.com/paper/2512.07344/) (2025) - [City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning](https://mlsanity.com/paper/2507.12795/) (2025) - [SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering](https://mlsanity.com/paper/2603.29437/) (2026) - [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025) - [Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification](https://mlsanity.com/paper/2604.16993/) (2026) - [Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering](https://mlsanity.com/paper/2608.28762/) (2026) ## Abstract (arXiv) 3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. While existing research has primarily focused on indoor household tasks and outdoor roadside autonomous driving tasks, there has been limited exploration of city-level scene understanding tasks. Furthermore, existing research faces challenges in understanding city scenes, due to the absence of spatial semantic information and human-environment interaction information at the city level.To address these challenges, we investigate 3D MQA from both dataset and method perspectives. From the dataset perspective, we introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding, which is the first dataset to incorporate scene semantic and human-environment interactive tasks within the city. From the method perspective, we propose a Scene graph enhanced City-level Understanding method (Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA methods and zero-shot using advanced large language models (LLMs), Sg-CityU demonstrates state-of-the-art (SOTA) performance in robustness and generalization.