# Temporal Action Localization by Structured Maximal Sums

arXiv: https://arxiv.org/abs/1704.04671 · HTML: https://mlsanity.com/paper/1704.04671/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2017.

## Built on by (most cited first, 28 of 28)

- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Object-Region Video Transformers](https://mlsanity.com/paper/2110.06915/) (2021)
- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [Activity Graph Transformer for Temporal Action Localization](https://mlsanity.com/paper/2101.08540/) (2021)
- [An Empirical Study of End-to-End Temporal Action Detection](https://mlsanity.com/paper/2204.02932/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Graph Convolutional Module for Temporal Action Localization in Videos](https://mlsanity.com/paper/2112.00302/) (2021)
- [Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2108.06524/) (2021)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization](https://mlsanity.com/paper/2103.15233/) (2021)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [Learning to Refactor Action and Co-occurrence Features for Temporal Action Localization](https://mlsanity.com/paper/2206.11493/) (2022)
- [Structured Context Transformer for Generic Event Boundary Detection](https://mlsanity.com/paper/2206.02985/) (2022)
- [Low-power, Continuous Remote Behavioral Localization with Event Cameras](https://mlsanity.com/paper/2312.03799/) (2023)
- [Weakly Supervised Temporal Anomaly Segmentation with Dynamic Time Warping](https://mlsanity.com/paper/2108.06816/) (2021)
- [SimOn: A Simple Framework for Online Temporal Action Localization](https://mlsanity.com/paper/2211.04905/) (2022)
- [SegTAD: Precise Temporal Action Detection via Semantic Segmentation](https://mlsanity.com/paper/2203.01542/) (2022)
- [Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning](https://mlsanity.com/paper/2206.11011/) (2022)
- [Local Compressed Video Stream Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2309.15431/) (2023)
- [Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/2206.10095/) (2022)
- [A Hybrid Graph Network for Complex Activity Detection in Video](https://mlsanity.com/paper/2310.17493/) (2023)
- [Efficient Action Counting with Dynamic Queries](https://mlsanity.com/paper/2403.01543/) (2024)
- [SADA: Semantic adversarial unsupervised domain adaptation for Temporal Action Localization](https://mlsanity.com/paper/2312.13377/) (2023)
- [DeepLocalization: Using change point detection for Temporal Action Localization](https://mlsanity.com/paper/2404.12258/) (2024)
- [Structured Context Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2512.00475/) (2025)
- [Dilation-Erosion for Single-Frame Supervised Temporal Action Localization](https://mlsanity.com/paper/2212.06348/) (2022)
- [Progression-Guided Temporal Action Detection in Videos](https://mlsanity.com/paper/2308.09268/) (2023)
- [Improving Weakly Supervised Temporal Action Localization by Exploiting Multi-resolution Information in Temporal Domain](https://mlsanity.com/paper/2506.18261/) (2025)

## Abstract (arXiv)

We address the problem of temporal action localization in videos. We pose action localization as a structured prediction over arbitrary-length temporal windows, where each window is scored as the sum of frame-wise classification scores. Additionally, our model classifies the start, middle, and end of each action as separate components, allowing our system to explicitly model each action's temporal evolution and take advantage of informative temporal dependencies present in this structure. In this framework, we localize actions by searching for the structured maximal sum, a problem for which we develop a novel, provably-efficient algorithmic solution. The frame-wise classification scores are computed using features from a deep Convolutional Neural Network (CNN), which are trained end-to-end to directly optimize for a novel structured objective. We evaluate our system on the THUMOS 14 action detection benchmark and achieve competitive performance.