# Representation Disparity-aware Distillation for 3D Object Detection

arXiv: https://arxiv.org/abs/2308.10308 · HTML: https://mlsanity.com/paper/2308.10308/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 40)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Learning Lightweight Lane Detection CNNs by Self Attention Distillation](https://mlsanity.com/paper/1908.00821/) (2019)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Pillar-based Object Detection for Autonomous Driving](https://mlsanity.com/paper/2007.10323/) (2020)
- [Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds](https://mlsanity.com/paper/2203.11139/) (2022)
- [Fast Point R-CNN](https://mlsanity.com/paper/1908.02990/) (2019)
- [RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection](https://mlsanity.com/paper/2106.13365/) (2021)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)

## Built on by (most cited first, 3 of 3)

- [RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features](https://mlsanity.com/paper/2403.05061/) (2024)
- [Reflectance Prediction-based Knowledge Distillation for Robust 3D Object Detection in Compressed Point Clouds](https://mlsanity.com/paper/2505.17442/) (2025)
- [Semantic-Supervised Spatial-Temporal Fusion for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2503.10579/) (2025)

## Abstract (arXiv)

In this paper, we focus on developing knowledge distillation (KD) for compact 3D detectors. We observe that off-the-shelf KD methods manifest their efficacy only when the teacher model and student counterpart share similar intermediate feature representations. This might explain why they are less effective in building extreme-compact 3D detectors where significant representation disparity arises due primarily to the intrinsic sparsity and irregularity in 3D point clouds. This paper presents a novel representation disparity-aware distillation (RDD) method to address the representation disparity issue and reduce performance gap between compact students and over-parameterized teachers. This is accomplished by building our RDD from an innovative perspective of information bottleneck (IB), which can effectively minimize the disparity of proposal region pairs from student and teacher in features and logits. Extensive experiments are performed to demonstrate the superiority of our RDD over existing KD methods. For example, our RDD increases mAP of CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance while taking up only 42% FLOPs.