# IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors

arXiv: https://arxiv.org/abs/2210.03477 · HTML: https://mlsanity.com/paper/2210.03477/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://mlsanity.com/paper/1602.02830/) (2016)
- [ConvMAE: Masked Convolution Meets Masked Autoencoders](https://mlsanity.com/paper/2205.03892/) (2022)
- [Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance](https://mlsanity.com/paper/1811.01335/) (2018)
- [Distilling Knowledge by Mimicking Features](https://mlsanity.com/paper/2011.01424/) (2020)
- [BiDet: An Efficient Binarized Object Detector](https://mlsanity.com/paper/2003.03961/) (2020)
- [Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation](https://mlsanity.com/paper/1811.12755/) (2018)
- [POEM: 1-bit Point-wise Operations based on Expectation-Maximization for Efficient Point Cloud Processing](https://mlsanity.com/paper/2111.13386/) (2021)
- [Cogradient Descent for Bilinear Optimization](https://mlsanity.com/paper/2006.09142/) (2020)

## Built on by (most cited first, 12 of 12)

- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [Q-DETR: An Efficient Low-Bit Quantized Detection Transformer](https://mlsanity.com/paper/2304.00253/) (2023)
- [Resilient Binary Neural Network](https://mlsanity.com/paper/2302.00956/) (2023)
- [Bi-ViT: Pushing the Limit of Vision Transformer Quantization](https://mlsanity.com/paper/2305.12354/) (2023)
- [BinaryViT: Towards Efficient and Accurate Binary Vision Transformers](https://mlsanity.com/paper/2305.14730/) (2023)
- [TTAQ: Towards Stable Post-training Quantization in Continuous Domain Adaptation](https://mlsanity.com/paper/2412.09899/) (2024)
- [Semantic RGB-D Image Synthesis](https://mlsanity.com/paper/2308.11356/) (2023)
- [In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Training](https://mlsanity.com/paper/2402.13497/) (2024)
- [Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review](https://mlsanity.com/paper/2508.03317/) (2025)
- [DCP-NAS: Discrepant Child-Parent Neural Architecture Search for 1-bit CNNs](https://mlsanity.com/paper/2306.15390/) (2023)
- [CycleBNN: Cyclic Precision Training in Binary Neural Networks](https://mlsanity.com/paper/2410.00050/) (2024)
- [Learning 1-Bit LiDAR-based Localization with Auxiliary Objective](https://mlsanity.com/paper/2606.27729/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) has been proven to be useful for training compact object detection models. However, we observe that KD is often effective when the teacher model and student counterpart share similar proposal information. This explains why existing KD methods are less effective for 1-bit detectors, caused by a significant information discrepancy between the real-valued teacher and the 1-bit student. This paper presents an Information Discrepancy-aware strategy (IDa-Det) to distill 1-bit detectors that can effectively eliminate information discrepancies and significantly reduce the performance gap between a 1-bit detector and its real-valued counterpart. We formulate the distillation process as a bi-level optimization formulation. At the inner level, we select the representative proposals with maximum information discrepancy. We then introduce a novel entropy distillation loss to reduce the disparity based on the selected proposals. Extensive experiments demonstrate IDa-Det's superiority over state-of-the-art 1-bit detectors and KD methods on both PASCAL VOC and COCO datasets. IDa-Det achieves a 76.9% mAP for a 1-bit Faster-RCNN with ResNet-18 backbone. Our code is open-sourced on https://github.com/SteveTsui/IDa-Det.