# TKD: Temporal Knowledge Distillation for Active Perception

arXiv: https://arxiv.org/abs/1903.01522 · HTML: https://mlsanity.com/paper/1903.01522/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 8 of 8)

- [Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation](https://mlsanity.com/paper/2203.06321/) (2022)
- [PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection](https://mlsanity.com/paper/2205.11098/) (2022)
- [Region-aware Knowledge Distillation for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2205.12451/) (2022)
- [Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection](https://mlsanity.com/paper/2211.08398/) (2022)
- [Streaming Video Analytics On The Edge With Asynchronous Cloud Support](https://mlsanity.com/paper/2210.01402/) (2022)
- [Temporal Knowledge Distillation for Time-Sensitive Financial Services Applications](https://mlsanity.com/paper/2312.16799/) (2023)
- [Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study](https://mlsanity.com/paper/2305.13232/) (2023)
- [Robust Salient Object Detection on Compressed Images Using Convolutional Neural Networks](https://mlsanity.com/paper/2409.13464/) (2024)

## Abstract (arXiv)

Deep neural networks based methods have been proved to achieve outstanding performance on object detection and classification tasks. Despite significant performance improvement, due to the deep structures, they still require prohibitive runtime to process images and maintain the highest possible performance for real-time applications. Observing the phenomenon that human vision system (HVS) relies heavily on the temporal dependencies among frames from the visual input to conduct recognition efficiently, we propose a novel framework dubbed as TKD: temporal knowledge distillation. This framework distills the temporal knowledge from a heavy neural networks based model over selected video frames (the perception of the moments) to a light-weight model. To enable the distillation, we put forward two novel procedures: 1) an Long-short Term Memory (LSTM) based key frame selection method; and 2) a novel teacher-bounded loss design. To validate, we conduct comprehensive empirical evaluations using different object detection methods over multiple datasets including Youtube-Objects and Hollywood scene dataset. Our results show consistent improvement in accuracy-speed trad-offs for object detection over the frames of the dynamic scene, compare to other modern object recognition methods.