mlsanityData to 7 Oct 2026

IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors

arXiv 2210.03477 ↗First posted 7 Oct 2022cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) has been proven to be useful for training compact object detection models. However, we observe that KD is often effective when the teacher model and student counterpart share similar proposal information. This explains why existing KD methods are less effective for 1-bit detectors, caused by a significant information discrepancy between the real-valued teacher and the 1-bit student. This paper presents an Information Discrepancy-aware strategy (IDa-Det) to distill 1-bit detectors that can effectively eliminate information discrepancies and significantly reduce the performance gap between a 1-bit detector and its real-valued counterpart. We formulate the distillation process as a bi-level optimization formulation. At the inner level, we select the representative proposals with maximum information discrepancy. We then introduce a novel entropy distillation loss to reduce the disparity based on the selected proposals. Extensive experiments demonstrate IDa-Det's superiority over state-of-the-art 1-bit detectors and KD methods on both PASCAL VOC and COCO datasets. IDa-Det achieves a 76.9% mAP for a 1-bit Faster-RCNN with ResNet-18 backbone. Our code is open-sourced on https://github.com/SteveTsui/IDa-Det.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

28 of 37 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  12. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
16 more
  1. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  2. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  3. 2013Do Deep Nets Really Need to be Deep?cited by 466
  4. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  5. 2013Predicting Parameters in Deep Learningcited by 223
  6. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  7. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  8. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  9. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  10. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  11. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  12. 2020Distilling Knowledge by Mimicking Featurescited by 34
  13. 2020BiDet: An Efficient Binarized Object Detectorcited by 33
  14. 2018Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagationcited by 29
  15. 2021POEM: 1-bit Point-wise Operations based on Expectation-Maximization for Efficient Point Cloud Processingcited by 6
  16. 2020Cogradient Descent for Bilinear Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 3; 2025: 1; 2026: 1.

12 papers.

  1. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  2. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 23
  3. 2023Resilient Binary Neural Networkcited by 13
  4. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantizationcited by 13
  5. 2023BinaryViT: Towards Efficient and Accurate Binary Vision Transformerscited by 7
  6. 2024TTAQ: Towards Stable Post-training Quantization in Continuous Domain Adaptationcited by 2
  7. 2023Semantic RGB-D Image Synthesiscited by 1
  8. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Trainingcited by 1
  9. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Reviewcited by 1
  10. 2023DCP-NAS: Discrepant Child-Parent Neural Architecture Search for 1-bit CNNscited by 0
  11. 2024CycleBNN: Cyclic Precision Training in Binary Neural Networkscited by 0
  12. 2026Learning 1-Bit LiDAR-based Localization with Auxiliary Objectivecited by 0