# Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy

arXiv: https://arxiv.org/abs/2501.18509 · HTML: https://mlsanity.com/paper/2501.18509/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [LocVTP: Video-Text Pre-training for Temporal Localization](https://mlsanity.com/paper/2207.10362/) (2022)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [Temporal Gaussian Mixture Layer for Videos](https://mlsanity.com/paper/1803.06316/) (2018)
- [Modeling Multi-Label Action Dependencies for Temporal Action Localization](https://mlsanity.com/paper/2103.03027/) (2021)
- [Learning Latent Super-Events to Detect Multiple Activities in Videos](https://mlsanity.com/paper/1712.01938/) (2017)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [Token Turing Machines](https://mlsanity.com/paper/2211.09119/) (2022)
- [PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points](https://mlsanity.com/paper/2210.11035/) (2022)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [Test-Time Zero-Shot Temporal Action Localization](https://mlsanity.com/paper/2404.05426/) (2024)
- [CTRN: Class-Temporal Relational Network for Action Detection](https://mlsanity.com/paper/2110.13473/) (2021)
- [TAN: Temporal Aggregation Network for Dense Multi-label Action Recognition](https://mlsanity.com/paper/1812.06203/) (2018)
- [PAT: Position-Aware Transformer for Dense Multi-Label Action Detection](https://mlsanity.com/paper/2308.05051/) (2023)
- [Contrastive Language-Action Pre-training for Temporal Localization](https://mlsanity.com/paper/2204.12293/) (2022)
- [PLOT-TAL: Prompt Learning with Optimal Transport for Few-Shot Temporal Action Localization](https://mlsanity.com/paper/2403.18915/) (2024)
- [AAN: Attributes-Aware Network for Temporal Action Detection](https://mlsanity.com/paper/2309.00696/) (2023)

## Abstract (arXiv)

Dense action detection involves detecting multiple co-occurring actions while action classes are often ambiguous and represent overlapping concepts. We argue that handling the dual challenge of temporal and class overlaps is too complex to effectively be tackled by a single network. To address this, we propose to decompose the task of detecting dense ambiguous actions into detecting dense, unambiguous sub-concepts that form the action classes (i.e., action entities and action motions), and assigning these sub-tasks to distinct sub-networks. By isolating these unambiguous concepts, the sub-networks can focus exclusively on resolving a single challenge, dense temporal overlaps. Furthermore, simultaneous actions in a video often exhibit interrelationships, and exploiting these relationships can improve the method performance. However, current dense action detection networks fail to effectively learn these relationships due to their reliance on binary cross-entropy optimization, which treats each class independently. To address this limitation, we propose providing explicit supervision on co-occurring concepts during network optimization through a novel language-guided contrastive learning loss. Our extensive experiments demonstrate the superiority of our approach over state-of-the-art methods, achieving substantial improvements of 3.8% and 1.7% on average across all metrics on the challenging benchmark datasets, Charades and MultiTHUMOS.