# Privileged Knowledge Distillation for Online Action Detection

arXiv: https://arxiv.org/abs/2011.09158 · HTML: https://mlsanity.com/paper/2011.09158/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 18 of 18)

- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [GateHUB: Gated History Unit with Background Suppression for Online Action Detection](https://mlsanity.com/paper/2206.04668/) (2022)
- [On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis](https://mlsanity.com/paper/2203.07976/) (2022)
- [Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models](https://mlsanity.com/paper/2203.16755/) (2022)
- [Continual Transformers: Redundancy-Free Attention for Online Inference](https://mlsanity.com/paper/2201.06268/) (2022)
- [Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos](https://mlsanity.com/paper/2203.13309/) (2022)
- [E2E-LOAD: End-to-End Long-form Online Action Detection](https://mlsanity.com/paper/2306.07703/) (2023)
- [JOADAA: joint online action detection and action anticipation](https://mlsanity.com/paper/2309.06130/) (2023)
- [Overlooked Poses Actually Make Sense: Distilling Privileged Knowledge for Human Motion Prediction](https://mlsanity.com/paper/2208.01302/) (2022)
- [A Circular Window-based Cascade Transformer for Online Action Detection](https://mlsanity.com/paper/2208.14209/) (2022)
- [Privileged Prior Information Distillation for Image Matting](https://mlsanity.com/paper/2211.14036/) (2022)
- [MALT: Multi-scale Action Learning Transformer for Online Action Detection](https://mlsanity.com/paper/2405.20892/) (2024)
- [O-TALC: Steps Towards Combating Oversegmentation within Online Action Segmentation](https://mlsanity.com/paper/2404.06894/) (2024)
- [Efficient Online Processing with Deep Neural Networks](https://mlsanity.com/paper/2306.13474/) (2023)
- [CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning](https://mlsanity.com/paper/2603.23988/) (2026)
- [How Much Future Helps? A Controlled Study of Future-Privileged Supervision for Causal Egocentric Gaze Estimation](https://mlsanity.com/paper/2607.01437/) (2026)

## Abstract (arXiv)

Online Action Detection (OAD) in videos is proposed as a per-frame labeling task to address the real-time prediction tasks that can only obtain the previous and current video frames. This paper presents a novel learning-with-privileged based framework for online action detection where the future frames only observable at the training stages are considered as a form of privileged information. Knowledge distillation is employed to transfer the privileged information from the offline teacher to the online student. We note that this setting is different from conventional KD because the difference between the teacher and student models mostly lies in input data rather than the network architecture. We propose Privileged Knowledge Distillation (PKD) which (i) schedules a curriculum learning procedure and (ii) inserts auxiliary nodes to the student model, both for shrinking the information gap and improving learning performance. Compared to other OAD methods that explicitly predict future frames, our approach avoids learning unpredictable unnecessary yet inconsistent visual contents and achieves state-of-the-art accuracy on two popular OAD benchmarks, TVSeries and THUMOS14.