mlsanityData to 14 Oct 2026

Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving

arXiv 2309.14491 ↗First posted 25 Sep 2023cs.CVcited by 37

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Closed-set 3D perception models trained on only a pre-defined set of object categories can be inadequate for safety critical applications such as autonomous driving where new object types can be encountered after deployment. In this paper, we present a multi-modal auto labeling pipeline capable of generating amodal 3D bounding boxes and tracklets for training models on open-set categories without 3D human labels. Our pipeline exploits motion cues inherent in point cloud sequences in combination with the freely available 2D image-text pairs to identify and track all traffic participants. Compared to the recent studies in this domain, which can only provide class-agnostic auto labels limited to moving objects, our method can handle both static and moving objects in the unsupervised manner and is able to output open-vocabulary semantic labels thanks to the proposed vision-language knowledge distillation. Experiments on the Waymo Open Dataset show that our approach outperforms the prior work by significant margins on various unsupervised 3D perception tasks.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 57 of its 64 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

57 of 64 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  4. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  5. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  6. 2021Grounded Language-Image Pre-trainingcited by 1,156
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  8. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  10. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  11. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
38 more
  1. 2017Focal Loss for Dense Object Detectioncited by 752
  2. 2022Language-driven Semantic Segmentationcited by 717
  3. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  4. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  5. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  6. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  8. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  9. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  10. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  11. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  12. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  13. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  14. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  15. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  16. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  17. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  18. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  19. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  20. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  21. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  22. 2021RangeDet:In Defense of Range View for LiDAR-based 3D Object Detectioncited by 140
  23. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  24. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
  25. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  26. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  27. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  28. 2019Fast Point R-CNNcited by 119
  29. 2021SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloudcited by 119
  30. 2021RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detectioncited by 115
  31. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  32. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  33. 2022SWFormer: Sparse Window Transformer for 3D Object Detection in Point Cloudscited by 92
  34. 2020HVNet: Hybrid Voxel Network for LiDAR Based 3D Object Detectioncited by 82
  35. 2020Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detectioncited by 78
  36. 2019Identifying Unknown Instances for Autonomous Drivingcited by 74
  37. 2016Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networkscited by 72
  38. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 18; 2026: 2.

37 papers.

Sort:

By area:

  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  2. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  3. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  5. 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 28
  6. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  7. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  8. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  9. 2025RefAV: Towards Planning-Centric Scenario Miningcited by 13
  10. 2024Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detectioncited by 9
  11. 2024Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detectioncited by 9
  12. 2024Lidar Panoptic Segmentation in an Open Worldcited by 8
25 more
  1. 2025LeAP: Consistent multi-domain 3D labeling using Foundation Modelscited by 8
  2. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Drivingcited by 8
  3. 2024Approaching Outside: Scaling Unsupervised 3D Object Detection from 2D Scenecited by 7
  4. 2025Tracking Meets Large Multimodal Models for Driving Scenario Understandingcited by 7
  5. 2025Towards Learning to Complete Anything in Lidarcited by 7
  6. 2024Radar Spectra-Language Model for Automotive Scene Parsingcited by 6
  7. 2025Zero-Shot 4D Lidar Panoptic Segmentationcited by 6
  8. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  9. 2025OpenBox: Annotate Any Bounding Boxes in 3Dcited by 6
  10. 20243D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentationcited by 5
  11. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 5
  12. 2024Learning 3D Perception from Others' Predictionscited by 4
  13. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  14. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  15. 2024Label-Efficient 3D Object Detection For Road-Side Unitscited by 3
  16. 2024Uncertainty-Guided Enhancement on Driving Perception System via Foundation Modelscited by 3
  17. 2024HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenescited by 2
  18. 2025Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Drivingcited by 2
  19. 2025OV-SCAN: Semantically Consistent Alignment for Novel Object Discovery in Open-Vocabulary 3D Object Detectioncited by 2
  20. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  21. 2026FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detectioncited by 2
  22. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  23. 2024Zero-shot 3D General Obstacle Detection via Multimodal Foundation Models and Geometrycited by 0
  24. 2025From Dataset to Real-world: General 3D Object Detection via Generalized Cross-domain Few-shot Learningcited by 0
  25. 2026ALOOD: Exploiting Language Representations for LiDAR-based Out-of-Distribution Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
  2. 2025VESPA: Towards un(Human)supervised Open-World Pointcloud Labeling for Autonomous Driving
  3. 2026Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models
  4. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
  5. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
  6. 2025OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving
  7. 20243D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving
  8. 2025CARScenes: Semantic VLM Dataset for Safe Autonomous Driving