mlsanityData to 14 Oct 2026

4D-Former: Multimodal 4D Panoptic Segmentation

arXiv 2311.01520 ↗First posted 2 Nov 2023cs.CV cs.ROcited by 12

Not on the map. Why?

Abstract (arXiv)

4D panoptic segmentation is a challenging but practically useful task that requires every point in a LiDAR point-cloud sequence to be assigned a semantic class label, and individual objects to be segmented and tracked over time. Existing approaches utilize only LiDAR inputs which convey limited information in regions with point sparsity. This problem can, however, be mitigated by utilizing RGB camera images which offer appearance-based information that can reinforce the geometry-based LiDAR features. Motivated by this, we propose 4D-Former: a novel method for 4D panoptic segmentation which leverages both LiDAR and image modalities, and predicts semantic masks as well as temporally consistent object masks for the input point-cloud sequence. We encode semantic classes and objects using a set of concise queries which absorb feature information from both data modalities. Additionally, we propose a learned mechanism to associate object tracks over time which reasons over both appearance and spatial location. We apply 4D-Former to the nuScenes and SemanticKITTI datasets where it achieves state-of-the-art results.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

46 of 57 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  9. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  10. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  11. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  12. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
34 more
  1. 2020PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentationcited by 275
  2. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  3. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  4. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  5. 2021RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentationcited by 173
  6. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  7. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 166
  8. 2021Multimodal Virtual Point 3D Detectioncited by 166
  9. 2021(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Networkcited by 164
  10. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  11. 2021Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentationcited by 107
  12. 2020LiDAR-based Panoptic Segmentation via Dynamic Shifting Networkcited by 84
  13. 2021EfficientLPS: Efficient LiDAR Panoptic Segmentationcited by 76
  14. 2020Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Drivingcited by 73
  15. 2022TorchSparse: Efficient Point Cloud Inference Enginecited by 73
  16. 2023MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Drivingcited by 68
  17. 2021Box-Aware Feature Enhancement for Single Object Tracking on Point Cloudscited by 66
  18. 20214D Panoptic LiDAR Segmentationcited by 65
  19. 2021EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentationcited by 59
  20. 2022Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmapcited by 59
  21. 2021Exploring Simple 3D Multi-Object Tracking for Autonomous Drivingcited by 56
  22. 2020MOPT: Multi-Object Panoptic Trackingcited by 53
  23. 2020Panoster: End-to-end Panoptic Segmentation of LiDAR Point Cloudscited by 46
  24. 2021Learnable Online Graph Representations for 3D Multi-Object Trackingcited by 38
  25. 2021GP-S3Net: Graph-based Panoptic Sparse Semantic Segmentation Networkcited by 37
  26. 20224D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregationcited by 35
  27. 2021HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Imagescited by 21
  28. 20234D Panoptic Segmentation as Invariant and Equivariant Field Predictioncited by 19
  29. 2023Position-Guided Point Cloud Panoptic Segmentation Transformercited by 17
  30. 2023PUPS: Point Cloud Unified Panoptic Segmentationcited by 16
  31. 2021CPSeg: Cluster-free Panoptic Segmentation of 3D LiDAR Point Cloudscited by 14
  32. 2021SMAC-Seg: LiDAR Panoptic Segmentation via Sparse Multi-directional Attention Clusteringcited by 11
  33. 2021A Divide-and-Merge Point Cloud Clustering Algorithm for LiDAR Panoptic Segmentationcited by 9
  34. 20217th AI Driving Olympics: 1st Place Report for Panoptic Trackingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2025DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentationcited by 31
  2. 2025How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentationcited by 8
  3. 2024ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Drivingcited by 6
  4. 2025Zero-Shot 4D Lidar Panoptic Segmentationcited by 6
  5. 2024Interactive4D: Interactive 4D LiDAR Segmentationcited by 4
  6. 2025nuScenes Revisited: Progress and Challenges in Autonomous Drivingcited by 2
  7. 2024MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motioncited by 1
  8. 2024Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detectioncited by 1
  9. 2024DQFormer: Towards Unified LiDAR Panoptic Segmentation with Decoupled Queriescited by 0
  10. 2025Training-Free Global Geometric Association for 4D LiDAR Panoptic Segmentationcited by 0
  11. 2026PanDA: Unsupervised Domain Adaptation for Multimodal 3D Panoptic Segmentation in Autonomous Drivingcited by 0
  12. 2026DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Drivingcited by 0