# Re-examining Distillation For Continual Object Detection

arXiv: https://arxiv.org/abs/2204.01407 · HTML: https://mlsanity.com/paper/2204.01407/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams](https://mlsanity.com/paper/2009.00919/) (2020)
- [Memory Efficient Experience Replay for Streaming Learning](https://mlsanity.com/paper/1809.05922/) (2018)
- [Meta-Learning with Warped Gradient Descent](https://mlsanity.com/paper/1909.00025/) (2019)
- [Optimal Continual Learning has Perfect Memory and is NP-hard](https://mlsanity.com/paper/2006.05188/) (2020)
- [Rehearsal revealed: The limits and merits of revisiting samples in continual learning](https://mlsanity.com/paper/2104.07446/) (2021)
- [Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN](https://mlsanity.com/paper/2003.03901/) (2020)
- [Incremental Object Detection via Meta-Learning](https://mlsanity.com/paper/2003.08798/) (2020)
- [RODEO: Replay for Online Object Detection](https://mlsanity.com/paper/2008.06439/) (2020)
- [Continual Learning for Domain Adaptation in Chest X-ray Classification](https://mlsanity.com/paper/2001.05922/) (2020)
- [Revisiting knowledge transfer for training object class detectors](https://mlsanity.com/paper/1708.06128/) (2017)
- [Lifelong Object Detection](https://mlsanity.com/paper/2009.01129/) (2020)

## Built on by (most cited first, 5 of 5)

- [Continual Detection Transformer for Incremental Object Detection](https://mlsanity.com/paper/2304.03110/) (2023)
- [Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection](https://mlsanity.com/paper/2307.12427/) (2023)
- [Object Detectors in the Open Environment: Challenges, Solutions, and Outlook](https://mlsanity.com/paper/2403.16271/) (2024)
- [3rd Continual Learning Workshop Challenge on Egocentric Category and Instance Level Object Understanding](https://mlsanity.com/paper/2212.06833/) (2022)
- [Continual Vision-and-Language Navigation](https://mlsanity.com/paper/2403.15049/) (2024)

## Abstract (arXiv)

Training models continually to detect and classify objects, from new classes and new domains, remains an open problem. In this work, we conduct a thorough analysis of why and how object detection models forget catastrophically. We focus on distillation-based approaches in two-stage networks; the most-common strategy employed in contemporary continual object detection work.Distillation aims to transfer the knowledge of a model trained on previous tasks -- the teacher -- to a new model -- the student -- while it learns the new task. We show that this works well for the region proposal network, but that wrong, yet overly confident teacher predictions prevent student models from effective learning of the classification head. Our analysis provides a foundation that allows us to propose improvements for existing techniques by detecting incorrect teacher predictions, based on current ground-truth labels, and by employing an adaptive Huber loss as opposed to the mean squared error for the distillation loss in the classification heads. We evidence that our strategy works not only in a class incremental setting, but also in domain incremental settings, which constitute a realistic context, likely to be the setting of representative real-world problems.