# Diagnosing Error in Temporal Action Detectors

arXiv: https://arxiv.org/abs/1807.10706 · HTML: https://mlsanity.com/paper/1807.10706/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 30 of 71)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [FineAction: A Fine-Grained Video Dataset for Temporal Action Localization](https://mlsanity.com/paper/2105.11107/) (2021)
- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2203.15187/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Ego-Only: Egocentric Action Detection without Exocentric Transferring](https://mlsanity.com/paper/2301.01380/) (2023)
- [Proposal-Free Temporal Action Detection via Global Segmentation Mask Learning](https://mlsanity.com/paper/2207.06580/) (2022)
- [BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection](https://mlsanity.com/paper/2205.02717/) (2022)
- [RCL: Recurrent Continuous Localization for Temporal Action Detection](https://mlsanity.com/paper/2203.07112/) (2022)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Weakly Supervised Action Selection Learning in Video](https://mlsanity.com/paper/2105.02439/) (2021)
- [Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization](https://mlsanity.com/paper/2106.14118/) (2021)
- [Background-Click Supervision for Temporal Action Localization](https://mlsanity.com/paper/2111.12449/) (2021)
- [MMAD: Multi-label Micro-Action Detection in Videos](https://mlsanity.com/paper/2407.05311/) (2024)
- [Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization](https://mlsanity.com/paper/2211.14053/) (2022)
- [A Survey on Video Moment Localization](https://mlsanity.com/paper/2306.07515/) (2023)
- [The Blessings of Unlabeled Background in Untrimmed Videos](https://mlsanity.com/paper/2103.13183/) (2021)
- [Adapting Short-Term Transformers for Action Detection in Untrimmed Videos](https://mlsanity.com/paper/2312.01897/) (2023)
- [TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression](https://mlsanity.com/paper/2404.02405/) (2024)
- [Temporal Action Localization with Enhanced Instant Discriminability](https://mlsanity.com/paper/2309.05590/) (2023)
- [FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation](https://mlsanity.com/paper/2402.00293/) (2024)
- [Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions](https://mlsanity.com/paper/2403.20254/) (2024)

## Abstract (arXiv)

Despite the recent progress in video understanding and the continuous rate of improvement in temporal action localization throughout the years, it is still unclear how far (or close?) we are to solving the problem. To this end, we introduce a new diagnostic tool to analyze the performance of temporal action detectors in videos and compare different methods beyond a single scalar metric. We exemplify the use of our tool by analyzing the performance of the top rewarded entries in the latest ActivityNet action localization challenge. Our analysis shows that the most impactful areas to work on are: strategies to better handle temporal context around the instances, improving the robustness w.r.t. the instance absolute and relative size, and strategies to reduce the localization errors. Moreover, our experimental analysis finds the lack of agreement among annotator is not a major roadblock to attain progress in the field. Our diagnostic tool is publicly available to keep fueling the minds of other researchers with additional insights about their algorithms.