mlsanityData to 14 Oct 2026

Zero-Shot 4D Lidar Panoptic Segmentation

arXiv 2504.00848 ↗First posted 1 Apr 2025cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Zero-shot 4D segmentation and recognition of arbitrary objects in Lidar is crucial for embodied navigation, with applications ranging from streaming perception to semantic mapping and localization. However, the primary challenge in advancing research and developing generalized, versatile methods for spatio-temporal scene understanding in Lidar lies in the scarcity of datasets that provide the necessary diversity and scale of annotations.To overcome these challenges, we propose SAL-4D (Segment Anything in Lidar--4D), a method that utilizes multi-modal robotic sensor setups as a bridge to distill recent developments in Video Object Segmentation (VOS) in conjunction with off-the-shelf Vision-Language foundation models to Lidar. We utilize VOS models to pseudo-label tracklets in short video sequences, annotate these tracklets with sequence-level CLIP tokens, and lift them to the 4D Lidar space using calibrated multi-modal sensory setups to distill them to our SAL-4D model. Due to temporal consistent predictions, we outperform prior art in 3D Zero-Shot Lidar Panoptic Segmentation (LPS) over 5 PQ, and unlock Zero-Shot 4D-LPS.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 87 of its 115 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

87 of 115 references.

References · 87

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  11. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  12. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
38 more
  1. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  2. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  3. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  4. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  6. 2018Panoptic Segmentationcited by 802
  7. 2022Language-driven Semantic Segmentationcited by 717
  8. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  9. 2021Extract Free Dense Labels from CLIPcited by 583
  10. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  11. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  12. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  13. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  14. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  15. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  16. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  17. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  18. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  19. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  20. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  21. 2019Zero-Shot Semantic Segmentationcited by 245
  22. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 232
  23. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 232
  24. 2018SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloudcited by 228
  25. 2014Towards Open World Recognitioncited by 226
  26. 2021Group-Free 3D Object Detection via Transformerscited by 215
  27. 2018Zero-Shot Object Detectioncited by 189
  28. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 166
  29. 2020MOTChallenge: A Benchmark for Single-Camera Multiple Target Trackingcited by 136
  30. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  31. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  32. 2015A Novel Performance Evaluation Methodology for Single-Target Trackerscited by 113
  33. 2017Dropout Sampling for Robust Object Detection in Open-Set Conditionscited by 112
  34. 2021Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentationcited by 107
  35. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  36. 2020LiDAR-based Panoptic Segmentation via Dynamic Shifting Networkcited by 84
  37. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 80
  38. 2021EfficientLPS: Efficient LiDAR Panoptic Segmentationcited by 76

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025LOSC: LiDAR Open-voc Segmentation Consolidatorcited by 5
  2. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  3. 2025BlendCLIP: Bridging Synthetic and Real Domains for Zero-Shot 3D Object Classification with Multimodal Pretrainingcited by 0
  4. 2025Training-Free Global Geometric Association for 4D LiDAR Panoptic Segmentationcited by 0
  5. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0
  6. 2026Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025B4DL: A Benchmark for 4D LiDAR LLM in Spatio-Temporal Understanding
  2. 2025Learning to Generate 4D LiDAR Sequences
  3. 2026Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers
  4. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
  5. 2026DENALI: A Dataset Enabling Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs
  6. 2023Few-shot 3D LiDAR Semantic Segmentation for Autonomous Driving
  7. 2025The Devil is in the Details: Simple Remedies for Image-to-LiDAR Representation Learning
  8. 20232D Feature Distillation for Weakly- and Semi-Supervised 3D Semantic Segmentation