# ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation

arXiv: https://arxiv.org/abs/2503.06307 · HTML: https://mlsanity.com/paper/2503.06307/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Channel-wise Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/2011.13256/) (2020)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [Knowledge Diffusion for Distillation](https://mlsanity.com/paper/2305.15712/) (2023)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Distilling Object Detectors with Feature Richness](https://mlsanity.com/paper/2111.00674/) (2021)
- [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023)
- [Masked Distillation with Receptive Tokens](https://mlsanity.com/paper/2205.14589/) (2022)
- [Prediction-Guided Distillation for Dense Object Detection](https://mlsanity.com/paper/2203.05469/) (2022)

## Built on by (most cited first, 18 of 18)

- [ReCo-KD: Region- and Context-Aware Knowledge Distillation for Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2601.08301/) (2026)
- [MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding](https://mlsanity.com/paper/2512.06581/) (2025)
- [From Performance to Practice: Knowledge-Distilled Segmentator for On-Premises Clinical Workflows](https://mlsanity.com/paper/2601.09191/) (2026)
- [Recursive Self-Evolving Agents via Held-Out Selection](https://mlsanity.com/paper/2606.28374/) (2026)
- [Pansharpening for Thin-Cloud Contaminated Remote Sensing Images: A Unified Framework and Benchmark Dataset](https://mlsanity.com/paper/2603.14952/) (2026)
- [Momentum Memory for Knowledge Distillation in Computational Pathology](https://mlsanity.com/paper/2602.21395/) (2026)
- [MIS-HCC: Hierarchical Channel Clustering for Efficient Medical Image Segmentation](https://mlsanity.com/paper/2607.17329/) (2026)
- [Self-Supervised Visual Prompting for Cross-Domain Road Damage Detection](https://mlsanity.com/paper/2511.12410/) (2025)
- [TDEC: Deep Embedded Image Clustering with Transformer and Distribution Information](https://mlsanity.com/paper/2603.26746/) (2026)
- [Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments](https://mlsanity.com/paper/2505.19699/) (2025)
- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentation](https://mlsanity.com/paper/2605.26382/) (2026)
- [CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filtering](https://mlsanity.com/paper/2603.26744/) (2026)
- [Deep Image Clustering Based on Curriculum Learning and Density Information](https://mlsanity.com/paper/2604.03306/) (2026)
- [Displacement Preserving Relational Distillation for Robust Medical Segmentation](https://mlsanity.com/paper/2607.04599/) (2026)
- [FlowTime: Towards Continuous Generative Watch Time Prediction via Flow-based Personalized Priors](https://mlsanity.com/paper/2606.01352/) (2026)
- [Two-Stage Cross-Domain Cervical Abnormality Screening with Cytopathological Image Synthesis and Knowledge Distillation](https://mlsanity.com/paper/2606.27678/) (2026)
- [Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement](https://mlsanity.com/paper/2609.19964/) (2026)

## Abstract (arXiv)

Dense visual prediction tasks, such as detection and segmentation, are crucial for time-critical applications (e.g., autonomous driving and video surveillance). While deep models achieve strong performance, their efficiency remains a challenge. Knowledge distillation (KD) is an effective model compression technique, but existing feature-based KD methods rely on static, teacher-driven feature selection, failing to adapt to the student's evolving learning state or leverage dynamic student-teacher interactions. To address these limitations, we propose Adaptive student-teacher Cooperative Attention Masking for Knowledge Distillation (ACAM-KD), which introduces two key components: (1) Student-Teacher Cross-Attention Feature Fusion (STCA-FF), which adaptively integrates features from both models for a more interactive distillation process, and (2) Adaptive Spatial-Channel Masking (ASCM), which dynamically generates importance masks to enhance both spatial and channel-wise feature selection. Unlike conventional KD methods, ACAM-KD adapts to the student's evolving needs throughout the entire distillation process. Extensive experiments on multiple benchmarks validate its effectiveness. For instance, on COCO2017, ACAM-KD improves object detection performance by up to 1.4 mAP over the state-of-the-art when distilling a ResNet-50 student from a ResNet-101 teacher. For semantic segmentation on Cityscapes, it boosts mIoU by 3.09 over the baseline with DeepLabV3-MobileNetV2 as the student model.