# CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning

arXiv: https://arxiv.org/abs/2603.23988 · HTML: https://mlsanity.com/paper/2603.23988/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [OadTR: Online Action Detection with Transformers](https://mlsanity.com/paper/2106.11149/) (2021)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Online Action Detection](https://mlsanity.com/paper/1604.06506/) (2016)
- [D3D: Distilled 3D Networks for Video Action Recognition](https://mlsanity.com/paper/1812.08249/) (2018)
- [Real-time Online Video Detection with Temporal Smoothing Transformers](https://mlsanity.com/paper/2209.09236/) (2022)
- [Omni-Dimensional Dynamic Convolution](https://mlsanity.com/paper/2209.07947/) (2022)
- [Learning to Discriminate Information for Online Action Detection](https://mlsanity.com/paper/1912.04461/) (2019)
- [TAda! Temporally-Adaptive Convolutions for Video Understanding](https://mlsanity.com/paper/2110.06178/) (2021)
- [GateHUB: Gated History Unit with Background Suppression for Online Action Detection](https://mlsanity.com/paper/2206.04668/) (2022)
- [Colar: Effective and Efficient Online Action Detection by Consulting Exemplars](https://mlsanity.com/paper/2203.01057/) (2022)
- [Privileged Knowledge Distillation for Online Action Detection](https://mlsanity.com/paper/2011.09158/) (2020)
- [LAP-Net: Adaptive Features Sampling via Learning Action Progression for Online Action Detection](https://mlsanity.com/paper/2011.07915/) (2020)
- [Temporally smooth online action detection using cycle-consistent future anticipation](https://mlsanity.com/paper/2104.08030/) (2021)
- [MALT: Multi-scale Action Learning Transformer for Online Action Detection](https://mlsanity.com/paper/2405.20892/) (2024)

## Abstract (arXiv)

Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.