# HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization

arXiv: https://arxiv.org/abs/2408.06437 · HTML: https://mlsanity.com/paper/2408.06437/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 30 of 36)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [Adaptive Attention Span in Transformers](https://mlsanity.com/paper/1905.07799/) (2019)
- [TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals](https://mlsanity.com/paper/1703.06189/) (2017)
- [Video Self-Stitching Graph Network for Temporal Action Localization](https://mlsanity.com/paper/2011.14598/) (2020)
- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [OadTR: Online Action Detection with Transformers](https://mlsanity.com/paper/2106.11149/) (2021)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Online Action Detection](https://mlsanity.com/paper/1604.06506/) (2016)
- [Enriching Local and Global Contexts for Temporal Action Localization](https://mlsanity.com/paper/2107.12960/) (2021)
- [Multi-shot Temporal Event Localization: a Benchmark](https://mlsanity.com/paper/2012.09434/) (2020)
- [Equalized Focal Loss for Dense Long-Tailed Object Detection](https://mlsanity.com/paper/2201.02593/) (2022)
- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [Learning to Discriminate Information for Online Action Detection](https://mlsanity.com/paper/1912.04461/) (2019)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [GateHUB: Gated History Unit with Background Suppression for Online Action Detection](https://mlsanity.com/paper/2206.04668/) (2022)
- [StartNet: Online Detection of Action Start in Untrimmed Videos](https://mlsanity.com/paper/1903.09868/) (2019)
- [Colar: Effective and Efficient Online Action Detection by Consulting Exemplars](https://mlsanity.com/paper/2203.01057/) (2022)
- [Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization](https://mlsanity.com/paper/2103.15233/) (2021)
- [Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization](https://mlsanity.com/paper/2211.14053/) (2022)

## Built on by (most cited first, 10 of 10)

- [Open-ended Hierarchical Streaming Video Understanding with Vision Language Models](https://mlsanity.com/paper/2509.12145/) (2025)
- [DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos](https://mlsanity.com/paper/2505.16376/) (2025)
- [Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding](https://mlsanity.com/paper/2508.04546/) (2025)
- [PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling](https://mlsanity.com/paper/2505.23155/) (2025)
- [REEF: Relevance-Aware and Efficient LLM Adapter for Video Understanding](https://mlsanity.com/paper/2504.05491/) (2025)
- [OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization](https://mlsanity.com/paper/2607.00289/) (2026)
- [SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusion](https://mlsanity.com/paper/2603.12764/) (2026)
- [OZ-TAL: Online Zero-Shot Temporal Action Localization](https://mlsanity.com/paper/2605.09976/) (2026)
- [Masked Diffusion Vision-Language Models for Temporal Action Localization](https://mlsanity.com/paper/2605.29858/) (2026)
- [NBA_Streaming: A Large-Scale Benchmark for Fine-Grained Basketball Commentary Generation in Continuous Streams](https://mlsanity.com/paper/2608.09200/) (2026)

## Abstract (arXiv)

Online video understanding often relies on individual frames, leading to frame-by-frame predictions. Recent advancements such as Online Temporal Action Localization (OnTAL), extend this approach to instance-level predictions. However, existing methods mainly focus on short-term context, neglecting historical information. To address this, we introduce the History-Augmented Anchor Transformer (HAT) Framework for OnTAL. By integrating historical context, our framework enhances the synergy between long-term and short-term information, improving the quality of anchor features crucial for classification and localization. We evaluate our model on both procedural egocentric (PREGO) datasets (EGTEA and EPIC) and standard non-PREGO OnTAL datasets (THUMOS and MUSES). Results show that our model outperforms state-of-the-art approaches significantly on PREGO datasets and achieves comparable or slightly superior performance on non-PREGO datasets, underscoring the importance of leveraging long-term history, especially in procedural and egocentric action scenarios. Code is available at: https://github.com/sakibreza/ECCV24-HAT/