mlsanityData to 14 Oct 2026

ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving

arXiv 2411.05311 ↗First posted 8 Nov 2024cs.CV cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Offboard perception aims to automatically generate high-quality 3D labels for autonomous driving (AD) scenes. Existing offboard methods focus on 3D object detection with closed-set taxonomy and fail to match human-level recognition capability on the rapidly evolving perception tasks. Due to heavy reliance on human labels and the prevalence of data imbalance and sparsity, a unified framework for offboard auto-labeling various elements in AD scenes that meets the distinct needs of perception tasks is not being fully explored. In this paper, we propose a novel multi-modal Zero-shot Offboard Panoptic Perception (ZOPP) framework for autonomous driving scenes. ZOPP integrates the powerful zero-shot recognition capabilities of vision foundation models and 3D representations derived from point clouds. To the best of our knowledge, ZOPP represents a pioneering effort in the domain of multi-modal panoptic perception and auto labeling for autonomous driving scenes. We conduct comprehensive empirical studies and evaluations on Waymo open dataset to validate the proposed ZOPP on various perception tasks. To further explore the usability and extensibility of our proposed ZOPP, we also conduct experiments in downstream applications. The results further demonstrate the great potential of our ZOPP for real-world scenarios.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 63 of its 73 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

63 of 73 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Segment Anythingcited by 8,234
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  10. 2020Point Transformercited by 1,470
  11. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
38 more
  1. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  3. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  5. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  6. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  7. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  8. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 410
  9. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  10. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  11. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 368
  12. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  13. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  14. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  15. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  16. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 207
  17. 2023Segment and Track Anythingcited by 193
  18. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  19. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 148
  20. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 145
  21. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  22. 2021SimpleTrack: Understanding and Rethinking 3D Multi-object Trackingcited by 103
  23. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 86
  24. 2020LiDAR-based Panoptic Segmentation via Dynamic Shifting Networkcited by 84
  25. 2023Open-Vocabulary Point-Cloud Object Detection without 3D Annotationcited by 81
  26. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  27. 2022LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perceptioncited by 74
  28. 20214D Panoptic LiDAR Segmentationcited by 65
  29. 2022MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detectioncited by 65
  30. 2022Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmapcited by 59
  31. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  32. 2021Semantic Scene Completion via Integrating Instances and Scene in-the-Loopcited by 55
  33. 2021Auto4D: Learning to Label 4D Objects from Sequential Point Cloudscited by 42
  34. 2021Immortal Tracker: Tracklet Never Diescited by 42
  35. 2023SAM3D: Zero-Shot 3D Object Detection via Segment Anything Modelcited by 40
  36. 2019RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Drivingcited by 39
  37. 2019FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Datacited by 33
  38. 2023DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Cloudscited by 31

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  2. 2025Event-aided Semantic Scene Completioncited by 3
  3. 2025Out-of-Distribution Semantic Occupancy Predictioncited by 2
  4. 2025ALISE: Annotation-Free LiDAR Instance Segmentation for Autonomous Drivingcited by 1
  5. 2025Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurancecited by 0
  6. 2026A Differentiable Optimization Framework for Registering Sequential Bounding Boxes with Point Cloud Streamcited by 0