# PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient

arXiv: https://arxiv.org/abs/2207.02039 · HTML: https://mlsanity.com/paper/2207.02039/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 41)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [TOOD: Task-aligned One-stage Object Detection](https://mlsanity.com/paper/2108.07755/) (2021)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)

## Built on by (most cited first, 30 of 41)

- [PromptKD: Unsupervised Prompt Distillation for Vision-Language Models](https://mlsanity.com/paper/2403.02781/) (2024)
- [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels](https://mlsanity.com/paper/2303.13005/) (2023)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [DAMO-YOLO : A Report on Real-Time Object Detection Design](https://mlsanity.com/paper/2211.15444/) (2022)
- [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023)
- [BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation](https://mlsanity.com/paper/2306.08075/) (2023)
- [AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2405.12944/) (2024)
- [Cloud-Device Collaborative Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2312.16279/) (2023)
- [LabelDistill: Label-guided Cross-modal Knowledge Distillation for Camera-based 3D Object Detection](https://mlsanity.com/paper/2407.10164/) (2024)
- [Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection](https://mlsanity.com/paper/2308.14286/) (2023)
- [MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation](https://mlsanity.com/paper/2409.18800/) (2024)
- [Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detection](https://mlsanity.com/paper/2503.02101/) (2025)
- [Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning](https://mlsanity.com/paper/2311.13934/) (2023)
- [Domain-invariant Progressive Knowledge Distillation for UAV-based Object Detection](https://mlsanity.com/paper/2408.11407/) (2024)
- [DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer](https://mlsanity.com/paper/2505.15133/) (2025)
- [Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.11646/) (2023)
- [Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Loss](https://mlsanity.com/paper/2409.17823/) (2024)
- [OD3: Optimization-free Dataset Distillation for Object Detection](https://mlsanity.com/paper/2506.01942/) (2025)
- [DiReDi: Distillation and Reverse Distillation for AIoT Applications](https://mlsanity.com/paper/2409.08308/) (2024)
- [SAMKD: Spatial-aware Adaptive Masking Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2501.07101/) (2025)
- [MonoSKD: General Distillation Framework for Monocular 3D Object Detection via Spearman Correlation Coefficient](https://mlsanity.com/paper/2310.11316/) (2023)
- [AI-based association analysis for medical imaging using latent-space geometric confounder correction](https://mlsanity.com/paper/2311.12836/) (2023)
- [Precise Knowledge Transfer via Flow Matching](https://mlsanity.com/paper/2402.02012/) (2024)
- [Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection](https://mlsanity.com/paper/2406.06999/) (2024)
- [SNN-PAR: Energy Efficient Pedestrian Attribute Recognition via Spiking Neural Networks](https://mlsanity.com/paper/2410.07857/) (2024)
- [Centerness-based Instance-aware Knowledge Distillation with Task-wise Mutual Lifting for Object Detection on Drone Imagery](https://mlsanity.com/paper/2411.02861/) (2024)
- [InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective](https://mlsanity.com/paper/2505.21920/) (2025)
- [Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review](https://mlsanity.com/paper/2508.03317/) (2025)
- [CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation](https://mlsanity.com/paper/2511.21503/) (2025)
- [Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictions](https://mlsanity.com/paper/2401.13174/) (2024)

## Abstract (arXiv)

Knowledge distillation(KD) is a widely-used technique to train compact models in object detection. However, there is still a lack of study on how to distill between heterogeneous detectors. In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different. However, directly aligning the feature maps to distill detectors suffers from two problems. First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student. Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise. To address the above issues, we propose to imitate features with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features. Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs. Furthermore, it converges faster. With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5% and 43.9% mAP on COCO2017, which are 4.1% and 4.8% higher than the baseline, respectively.