# Learning Latent Super-Events to Detect Multiple Activities in Videos

arXiv: https://arxiv.org/abs/1712.01938 · HTML: https://mlsanity.com/paper/1712.01938/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2017.

## Built on by (most cited first, 30 of 30)

- [Language Repository for Long Video Understanding](https://mlsanity.com/paper/2403.14622/) (2024)
- [Activity Graph Transformer for Temporal Action Localization](https://mlsanity.com/paper/2101.08540/) (2021)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [Modeling Multi-Label Action Dependencies for Temporal Action Localization](https://mlsanity.com/paper/2103.03027/) (2021)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [Token Turing Machines](https://mlsanity.com/paper/2211.09119/) (2022)
- [PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points](https://mlsanity.com/paper/2210.11035/) (2022)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [Micro-gesture Online Recognition using Learnable Query Points](https://mlsanity.com/paper/2407.04490/) (2024)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [CTRN: Class-Temporal Relational Network for Action Detection](https://mlsanity.com/paper/2110.13473/) (2021)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [PAT: Position-Aware Transformer for Dense Multi-Label Action Detection](https://mlsanity.com/paper/2308.05051/) (2023)
- [Weakly-guided Self-supervised Pretraining for Temporal Activity Detection](https://mlsanity.com/paper/2111.13675/) (2021)
- [Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning](https://mlsanity.com/paper/2206.11011/) (2022)
- [Decoupled Spatial Temporal Graphs for Generic Visual Grounding](https://mlsanity.com/paper/2103.10191/) (2021)
- [PAMI-AD: An Activity Detector Exploiting Part-attention and Motion Information in Surveillance Videos](https://mlsanity.com/paper/2203.03796/) (2022)
- [A Multi-Modal Transformer Network for Action Detection](https://mlsanity.com/paper/2305.19624/) (2023)
- [MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videos](https://mlsanity.com/paper/2501.06138/) (2025)
- [NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition](https://mlsanity.com/paper/2104.01110/) (2021)
- [Co-Occurrence Matters: Learning Action Relation for Temporal Action Localization](https://mlsanity.com/paper/2303.08463/) (2023)
- [AAN: Attributes-Aware Network for Temporal Action Detection](https://mlsanity.com/paper/2309.00696/) (2023)
- [CLTA: Contents and Length-based Temporal Attention for Few-shot Action Recognition](https://mlsanity.com/paper/2103.10567/) (2021)
- [Low Pass Filter for Anti-aliasing in Temporal Action Localization](https://mlsanity.com/paper/2104.11403/) (2021)
- [An Effective-Efficient Approach for Dense Multi-Label Action Detection](https://mlsanity.com/paper/2406.06187/) (2024)
- [Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy](https://mlsanity.com/paper/2501.18509/) (2025)
- [VISTA: Dense Multi-Label Classroom Coding with Vision-Language Models](https://mlsanity.com/paper/2609.04550/) (2026)

## Abstract (arXiv)

In this paper, we introduce the concept of learning latent super-events from activity videos, and present how it benefits activity detection in continuous videos. We define a super-event as a set of multiple events occurring together in videos with a particular temporal organization; it is the opposite concept of sub-events. Real-world videos contain multiple activities and are rarely segmented (e.g., surveillance videos), and learning latent super-events allows the model to capture how the events are temporally related in videos. We design temporal structure filters that enable the model to focus on particular sub-intervals of the videos, and use them together with a soft attention mechanism to learn representations of latent super-events. Super-event representations are combined with per-frame or per-segment CNNs to provide frame-level annotations. Our approach is designed to be fully differentiable, enabling end-to-end learning of latent super-event representations jointly with the activity detector using them. Our experiments with multiple public video datasets confirm that the proposed concept of latent super-event learning significantly benefits activity detection, advancing the state-of-the-arts.