# Colar: Effective and Efficient Online Action Detection by Consulting Exemplars

arXiv: https://arxiv.org/abs/2203.01057 · HTML: https://mlsanity.com/paper/2203.01057/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 68 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Memory Enhanced Global-Local Aggregation for Video Object Detection](https://mlsanity.com/paper/2003.12063/) (2020)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Weakly Supervised Object Localization and Detection: A Survey](https://mlsanity.com/paper/2104.07918/) (2021)

## Built on by (most cited first, 21 of 21)

- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [Unified Transformer Tracker for Object Tracking](https://mlsanity.com/paper/2203.15175/) (2022)
- [HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization](https://mlsanity.com/paper/2408.06437/) (2024)
- [Learning Streaming Video Representation via Multitask Training](https://mlsanity.com/paper/2504.20041/) (2025)
- [E2E-LOAD: End-to-End Long-form Online Action Detection](https://mlsanity.com/paper/2306.07703/) (2023)
- [Online Temporal Action Localization with Memory-Augmented Transformer](https://mlsanity.com/paper/2408.02957/) (2024)
- [Structured Attention Composition for Temporal Action Localization](https://mlsanity.com/paper/2205.09956/) (2022)
- [Open-ended Hierarchical Streaming Video Understanding with Vision Language Models](https://mlsanity.com/paper/2509.12145/) (2025)
- [NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understanding](https://mlsanity.com/paper/2310.08326/) (2023)
- [A Circular Window-based Cascade Transformer for Online Action Detection](https://mlsanity.com/paper/2208.14209/) (2022)
- [ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos](https://mlsanity.com/paper/2407.12987/) (2024)
- [HiMemFormer: Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipation](https://mlsanity.com/paper/2411.01455/) (2024)
- [Track-On2: Enhancing Online Point Tracking with Memory](https://mlsanity.com/paper/2509.19115/) (2025)
- [MALT: Multi-scale Action Learning Transformer for Online Action Detection](https://mlsanity.com/paper/2405.20892/) (2024)
- [Text-driven Online Action Detection](https://mlsanity.com/paper/2501.13518/) (2025)
- [Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge](https://mlsanity.com/paper/2506.06174/) (2025)
- [GaTector+: A Unified Head-free Framework for Gaze Object and Gaze Following Prediction](https://mlsanity.com/paper/2510.25301/) (2025)
- [Driving behavior recognition via self-discovery learning](https://mlsanity.com/paper/2503.14194/) (2025)
- [Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding](https://mlsanity.com/paper/2510.10682/) (2025)
- [CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning](https://mlsanity.com/paper/2603.23988/) (2026)
- [OZ-TAL: Online Zero-Shot Temporal Action Localization](https://mlsanity.com/paper/2605.09976/) (2026)

## Abstract (arXiv)

Online action detection has attracted increasing research interests in recent years. Current works model historical dependencies and anticipate the future to perceive the action evolution within a video segment and improve the detection accuracy. However, the existing paradigm ignores category-level modeling and does not pay sufficient attention to efficiency. Considering a category, its representative frames exhibit various characteristics. Thus, the category-level modeling can provide complimentary guidance to the temporal dependencies modeling. This paper develops an effective exemplar-consultation mechanism that first measures the similarity between a frame and exemplary frames, and then aggregates exemplary features based on the similarity weights. This is also an efficient mechanism, as both similarity measurement and feature aggregation require limited computations. Based on the exemplar-consultation mechanism, the long-term dependencies can be captured by regarding historical frames as exemplars, while the category-level modeling can be achieved by regarding representative frames from a category as exemplars. Due to the complementarity from the category-level modeling, our method employs a lightweight architecture but achieves new high performance on three benchmarks. In addition, using a spatio-temporal network to tackle video frames, our method makes a good trade-off between effectiveness and efficiency. Code is available at https://github.com/VividLe/Online-Action-Detection.