# TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localization

arXiv: https://arxiv.org/abs/2303.09055 · HTML: https://mlsanity.com/paper/2303.09055/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 69 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)

## Built on by (most cited first, 28 of 28)

- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [360+x: A Panoptic Multi-modal Scene Understanding Dataset](https://mlsanity.com/paper/2404.00989/) (2024)
- [Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2404.09263/) (2024)
- [MMAD: Multi-label Micro-Action Detection in Videos](https://mlsanity.com/paper/2407.05311/) (2024)
- [DyFADet: Dynamic Feature Aggregation for Temporal Action Detection](https://mlsanity.com/paper/2407.03197/) (2024)
- [Temporal Action Localization with Enhanced Instant Discriminability](https://mlsanity.com/paper/2309.05590/) (2023)
- [Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions](https://mlsanity.com/paper/2403.20254/) (2024)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [OpenTAD: A Unified Framework and Comprehensive Study of Temporal Action Detection](https://mlsanity.com/paper/2502.20361/) (2025)
- [Harnessing Temporal Causality for Advanced Temporal Action Detection](https://mlsanity.com/paper/2407.17792/) (2024)
- [Open-Vocabulary Action Localization with Iterative Visual Prompting](https://mlsanity.com/paper/2408.17422/) (2024)
- [Temporal Action Detection Model Compression by Progressive Block Drop](https://mlsanity.com/paper/2503.16916/) (2025)
- [LoSA: Long-Short-range Adapter for Scaling End-to-End Temporal Action Localization](https://mlsanity.com/paper/2404.01282/) (2024)
- [Zero-shot Action Localization via the Confidence of Large Vision-Language Models](https://mlsanity.com/paper/2410.14340/) (2024)
- [MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detection](https://mlsanity.com/paper/2511.17929/) (2025)
- [Spatial-Temporal Decoupled Adapter for Micro-gesture Online Recognition](https://mlsanity.com/paper/2606.07355/) (2026)
- [Boundary Discretization and Reliable Classification Network for Temporal Action Detection](https://mlsanity.com/paper/2310.06403/) (2023)
- [ContextDet: Temporal Action Detection with Adaptive Context Aggregation](https://mlsanity.com/paper/2410.15279/) (2024)
- [Benchmarking Data Efficiency and Computational Efficiency of Temporal Action Localization Models](https://mlsanity.com/paper/2308.13082/) (2023)
- [Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization](https://mlsanity.com/paper/2310.03456/) (2023)
- [LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation](https://mlsanity.com/paper/2604.18274/) (2026)
- [FDDet: Frequency-Decoupling for Boundary Refinement in Temporal Action Detection](https://mlsanity.com/paper/2504.00647/) (2025)
- [TBT-Former: Learning Temporal Boundary Distributions for Action Localization](https://mlsanity.com/paper/2512.01298/) (2025)
- [Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection](https://mlsanity.com/paper/2604.09164/) (2026)
- [Where are they looking in the operating room?](https://mlsanity.com/paper/2604.20574/) (2026)
- [Masked Diffusion Vision-Language Models for Temporal Action Localization](https://mlsanity.com/paper/2605.29858/) (2026)
- [SpikeTAD: Spiking Neural Networks for End-to-End Temporal Action Detection](https://mlsanity.com/paper/2606.12033/) (2026)
- [SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos](https://mlsanity.com/paper/2606.31127/) (2026)

## Abstract (arXiv)

Temporal Action Localization (TAL) is a challenging task in video understanding that aims to identify and localize actions within a video sequence. Recent studies have emphasized the importance of applying long-term temporal context modeling (TCM) blocks to the extracted video clip features such as employing complex self-attention mechanisms. In this paper, we present the simplest method ever to address this task and argue that the extracted video clip features are already informative to achieve outstanding performance without sophisticated architectures. To this end, we introduce TemporalMaxer, which minimizes long-term temporal context modeling while maximizing information from the extracted video clip features with a basic, parameter-free, and local region operating max-pooling block. Picking out only the most critical information for adjacent and local clip embeddings, this block results in a more efficient TAL model. We demonstrate that TemporalMaxer outperforms other state-of-the-art methods that utilize long-term TCM such as self-attention on various TAL datasets while requiring significantly fewer parameters and computational resources. The code for our approach is publicly available at https://github.com/TuanTNG/TemporalMaxer