# Open 3D World in Autonomous Driving

arXiv: https://arxiv.org/abs/2408.10880 · HTML: https://mlsanity.com/paper/2408.10880/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 39 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 5 of 5)

- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes](https://mlsanity.com/paper/2403.19589/) (2024)
- [FSD V2: Improving Fully Sparse 3D Object Detection with Virtual Voxels](https://mlsanity.com/paper/2308.03755/) (2023)
- [SSFlowNet: Semi-supervised Scene Flow Estimation On Point Clouds With Pseudo Label](https://mlsanity.com/paper/2312.15271/) (2023)
- [CMU-Flownet: Exploring Point Cloud Scene Flow Estimation in Occluded Scenario](https://mlsanity.com/paper/2404.10571/) (2024)

## Built on by (most cited first, 3 of 3)

- [Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities](https://mlsanity.com/paper/2509.08302/) (2025)
- [You Only Speak Once to See](https://mlsanity.com/paper/2409.18372/) (2024)
- [Position-aware Guided Point Cloud Completion with CLIP Model](https://mlsanity.com/paper/2412.08271/) (2024)

## Abstract (arXiv)

The capability for open vocabulary perception represents a significant advancement in autonomous driving systems, facilitating the comprehension and interpretation of a wide array of textual inputs in real-time. Despite extensive research in open vocabulary tasks within 2D computer vision, the application of such methodologies to 3D environments, particularly within large-scale outdoor contexts, remains relatively underdeveloped. This paper presents a novel approach that integrates 3D point cloud data, acquired from LIDAR sensors, with textual information. The primary focus is on the utilization of textual data to directly localize and identify objects within the autonomous driving context. We introduce an efficient framework for the fusion of bird's-eye view (BEV) region features with textual features, thereby enabling the system to seamlessly adapt to novel textual inputs and enhancing the robustness of open vocabulary detection tasks. The effectiveness of the proposed methodology is rigorously evaluated through extensive experimentation on the newly introduced NuScenes-T dataset, with additional validation of its zero-shot performance on the Lyft Level 5 dataset. This research makes a substantive contribution to the advancement of autonomous driving technologies by leveraging multimodal data to enhance open vocabulary perception in 3D environments, thereby pushing the boundaries of what is achievable in autonomous navigation and perception.