mlsanityData to 14 Oct 2026

Towards Efficient 3D Object Detection with Knowledge Distillation

arXiv 2205.15156 ↗First posted 30 May 2022cs.CV cs.AI cs.LGcited by 29

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

Despite substantial progress in 3D object detection, advanced 3D detectors often suffer from heavy computation overheads. To this end, we explore the potential of knowledge distillation (KD) for developing efficient 3D object detectors, focusing on popular pillar- and voxel-based detectors.In the absence of well-developed teacher-student pairs, we first study how to obtain student models with good trade offs between accuracy and efficiency from the perspectives of model compression and input resolution reduction. Then, we build a benchmark to assess existing KD methods developed in the 2D domain for 3D object detection upon six well-constructed teacher-student pairs. Further, we propose an improved KD pipeline incorporating an enhanced logit KD method that performs KD on only a few pivotal positions determined by teacher classification response, and a teacher-guided student model initialization to facilitate transferring teacher model's feature extraction ability to students through weight inheritance. Finally, we conduct extensive experiments on the Waymo dataset. Our best performing model achieves 65.75% LEVEL 2 mAPH, surpassing its teacher model and requiring only 44% of teacher flops. Our most efficient model runs 51 FPS on an NVIDIA A100, which is 2.2× faster than PointPillar with even higher accuracy. Code is available at urlhttps://github.com/CVMI-Lab/SparseKD.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 54 of its 72 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

54 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Focal and Global Knowledge Distillation for Detectors23 Nov
  2. 2021Multi-Scale Aligned Distillation for Low-Resolution Detection14 Sep
  3. 2021General Instance Distillation for Object Detection3 Mar

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2017Mask R-CNNcited by 7,277
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  11. 2014Going Deeper with Convolutionscited by 4,241
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  3. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  5. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  7. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  9. 2020Designing Network Design Spacescited by 960
  10. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  11. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  13. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  14. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 410
  15. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  16. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  17. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  18. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  19. 2017Submanifold Sparse Convolutional Networkscited by 223
  20. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 189
  21. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
  22. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  23. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  24. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
  25. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  26. 2022Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Cloudscited by 131
  27. 2019Fast Point R-CNNcited by 119
  28. 2021RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detectioncited by 115
  29. 2020EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruningcited by 80
  30. 2020Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detectioncited by 78
  31. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  32. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 70
  33. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  34. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 48
  35. 2020RangeRCNN: Towards Fast and Accurate 3D Object Detection with Range Image Representationcited by 37
  36. 2021Improving Object Detection by Label Assignment Distillationcited by 17
  37. 2020Fast Neural Network Adaptation via Parameter Remapping and Architecture Searchcited by 16
  38. 2021Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classificationcited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 10; 2025: 5; 2026: 2.

29 papers.

Sort:

By area:

  1. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  2. 2023SCPNet: Semantic Scene Completion on Point Cloudcited by 80
  3. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  4. 2023X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detectioncited by 28
  5. 2024RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Featurescited by 28
  6. 2024CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillationcited by 28
  7. 2025Perspective-Invariant 3D Object Detectioncited by 18
  8. 2022itKD: Interchange Transfer-based Knowledge Distillation for 3D Object Detectioncited by 14
  9. 2024TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentationcited by 13
  10. 2024Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detectioncited by 11
  11. 2025SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillationcited by 9
  12. 2023Geometric-aware Pretraining for Vision-centric 3D Object Detectioncited by 8
17 more
  1. 2023StereoDistill: Pick the Cream from LiDAR for Distilling Stereo-based 3D Object Detectioncited by 7
  2. 2024Weak-to-Strong 3D Object Detection with X-Ray Distillationcited by 5
  3. 2024PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interestcited by 4
  4. 2024Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillationcited by 4
  5. 2022Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detectioncited by 3
  6. 2023Representation Disparity-aware Distillation for 3D Object Detectioncited by 3
  7. 2023LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibrationcited by 3
  8. 2024LEROjD: Lidar Extended Radar-Only Object Detectioncited by 3
  9. 2025Reflectance Prediction-based Knowledge Distillation for Robust 3D Object Detection in Compressed Point Cloudscited by 3
  10. 2023Radio2Text: Streaming Speech Recognition Using mmWave Radio Signalscited by 1
  11. 2023Generalized Robot 3D Vision-Language Model with Fast Rendering and Pre-Training Vision-Language Alignmentcited by 1
  12. 2024MonoDETRNext: Next-Generation Accurate and Efficient Monocular 3D Object Detectorcited by 1
  13. 2025Semantic-Supervised Spatial-Temporal Fusion for LiDAR-based 3D Object Detectioncited by 1
  14. 2023Smaller3d: Smaller Models for 3D Semantic Segmentation Using Minkowski Engine and Knowledge Distillation Methodscited by 0
  15. 2025IMKD: Intensity-Aware Multi-Level Knowledge Distillation for Camera-Radar Fusioncited by 0
  16. 2026Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentationcited by 0
  17. 2026Lightweight 3D Object Detection via Mamba-Based Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review
  2. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
  3. 2022Prediction-Guided Distillation for Dense Object Detection
  4. 2022HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors
  5. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection
  6. 2023Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation
  7. 2021Revisiting Knowledge Distillation for Object Detection
  8. 2024Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection