# Reducing Capacity Gap in Knowledge Distillation with Review Mechanism for Crowd Counting

arXiv: https://arxiv.org/abs/2206.05475 · HTML: https://mlsanity.com/paper/2206.05475/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 11 Jun 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds](https://mlsanity.com/paper/1808.01050/) (2018)
- [Bayesian Loss for Crowd Count Estimation with Point Supervision](https://mlsanity.com/paper/1908.03684/) (2019)
- [Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detection](https://mlsanity.com/paper/1906.07538/) (2019)
- [Crowd Counting and Density Estimation by Trellis Encoder-Decoder Network](https://mlsanity.com/paper/1903.00853/) (2019)
- [HA-CCN: Hierarchical Attention-based Crowd Counting Network](https://mlsanity.com/paper/1907.10255/) (2019)
- [Shallow Feature Based Dense Attention Network for Crowd Counting](https://mlsanity.com/paper/2006.09853/) (2020)
- [ResKD: Residual-Guided Knowledge Distillation](https://mlsanity.com/paper/2006.04719/) (2020)
- [Encoder-Decoder Based Convolutional Neural Networks with Multi-Scale-Aware Modules for Crowd Counting](https://mlsanity.com/paper/2003.05586/) (2020)
- [Crowd Counting via Perspective-Guided Fractional-Dilation Convolution](https://mlsanity.com/paper/2107.03665/) (2021)
- [Efficient Crowd Counting via Structured Knowledge Transfer](https://mlsanity.com/paper/2003.10120/) (2020)
- [Residual Knowledge Distillation](https://mlsanity.com/paper/2002.09168/) (2020)
- [Crowd counting with crowd attention convolutional neural network](https://mlsanity.com/paper/2204.07347/) (2022)
- [Interlayer and Intralayer Scale Aggregation for Scale-invariant Crowd Counting](https://mlsanity.com/paper/2005.11943/) (2020)

## Abstract (arXiv)

The lightweight crowd counting models, in particular knowledge distillation (KD) based models, have attracted rising attention in recent years due to their superiority on computational efficiency and hardware requirement. However, existing KD based models usually suffer from the capacity gap issue, resulting in the performance of the student network being limited by the teacher network. In this paper, we address this issue by introducing a novel review mechanism following KD models, motivated by the review mechanism of human-beings during the study. Thus, the proposed model is dubbed ReviewKD. The proposed model consists of an instruction phase and a review phase, where we firstly exploit a well-trained heavy teacher network to transfer its latent feature to a lightweight student network in the instruction phase, then in the review phase yield a refined estimate of the density map based on the learned feature through a review mechanism. The effectiveness of ReviewKD is demonstrated by a set of experiments over six benchmark datasets via comparing to the state-of-the-art models. Numerical results show that ReviewKD outperforms existing lightweight models for crowd counting, and can effectively alleviate the capacity gap issue, and particularly has the performance beyond the teacher network. Besides the lightweight models, we also show that the suggested review mechanism can be used as a plug-and-play module to further boost the performance of a kind of heavy crowd counting models without modifying the neural network architecture and introducing any additional model parameter.