mlsanityData to 14 Oct 2026

Representation Disparity-aware Distillation for 3D Object Detection

arXiv 2308.10308 ↗First posted 20 Aug 2023cs.CVcited by 3

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

In this paper, we focus on developing knowledge distillation (KD) for compact 3D detectors. We observe that off-the-shelf KD methods manifest their efficacy only when the teacher model and student counterpart share similar intermediate feature representations. This might explain why they are less effective in building extreme-compact 3D detectors where significant representation disparity arises due primarily to the intrinsic sparsity and irregularity in 3D point clouds. This paper presents a novel representation disparity-aware distillation (RDD) method to address the representation disparity issue and reduce performance gap between compact students and over-parameterized teachers. This is accomplished by building our RDD from an innovative perspective of information bottleneck (IB), which can effectively minimize the disparity of proposal region pairs from student and teacher in features and logits. Extensive experiments are performed to demonstrate the superiority of our RDD over existing KD methods. For example, our RDD increases mAP of CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance while taking up only 42% FLOPs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 40 of its 50 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

40 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Towards Efficient 3D Object Detection with Knowledge Distillation30 May
  2. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection23 May
  3. 2021Focal and Global Knowledge Distillation for Detectors23 Nov
  4. 2021Multi-Scale Aligned Distillation for Low-Resolution Detection14 Sep
  5. 2021General Instance Distillation for Object Detection3 Mar

Other references · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2017Mask R-CNNcited by 7,277
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  8. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  11. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
23 more
  1. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  2. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  3. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  4. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  5. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 410
  6. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  7. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  8. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  9. 2017Submanifold Sparse Convolutional Networkscited by 223
  10. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 189
  11. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
  12. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  13. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  14. 2022Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Cloudscited by 131
  15. 2019Fast Point R-CNNcited by 119
  16. 2021RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detectioncited by 115
  17. 2020Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detectioncited by 78
  18. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  19. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  20. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  21. 2020Distilling Object Detectors with Task Adaptive Regularizationcited by 36
  22. 2020BiDet: An Efficient Binarized Object Detectorcited by 33
  23. 2021Improving Object Detection by Label Assignment Distillationcited by 17

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Featurescited by 28
  2. 2025Reflectance Prediction-based Knowledge Distillation for Robust 3D Object Detection in Compressed Point Cloudscited by 3
  3. 2025Semantic-Supervised Spatial-Temporal Fusion for LiDAR-based 3D Object Detectioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review
  2. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
  3. 2025Distilling Future Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection
  4. 2022Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection
  5. 2022Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection
  6. 2022Prediction-Guided Distillation for Dense Object Detection
  7. 2023StereoDistill: Pick the Cream from LiDAR for Distilling Stereo-based 3D Object Detection
  8. 2021Dynamic Knowledge Distillation With Noise Elimination for RGB-D Salient Object Detection