# A Perceptual Prediction Framework for Self Supervised Event Segmentation

arXiv: https://arxiv.org/abs/1811.04869 · HTML: https://mlsanity.com/paper/1811.04869/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2018.

## Built on by (most cited first, 30 of 40)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Action Shuffle Alternating Learning for Unsupervised Action Segmentation](https://mlsanity.com/paper/2104.02116/) (2021)
- [Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities](https://mlsanity.com/paper/2105.00067/) (2021)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [Temporal Action Segmentation with High-level Complex Activity Labels](https://mlsanity.com/paper/2108.06706/) (2021)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [UnweaveNet: Unweaving Activity Stories](https://mlsanity.com/paper/2112.10194/) (2021)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation](https://mlsanity.com/paper/2309.06276/) (2023)
- [Leveraging triplet loss for unsupervised action segmentation](https://mlsanity.com/paper/2304.06403/) (2023)
- [CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation](https://mlsanity.com/paper/2109.15170/) (2021)
- [Rethinking the Architecture Design for Efficient Generic Event Boundary Detection](https://mlsanity.com/paper/2407.12622/) (2024)
- [U-LanD: Uncertainty-Driven Video Landmark Detection](https://mlsanity.com/paper/2102.01586/) (2021)
- [MHMS: Multimodal Hierarchical Multimedia Summarization](https://mlsanity.com/paper/2204.03734/) (2022)
- [LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos](https://mlsanity.com/paper/2210.05840/) (2022)
- [CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2507.03539/) (2025)
- [Towards Active Vision for Action Localization with Reactive Control and Predictive Learning](https://mlsanity.com/paper/2111.05448/) (2021)
- [Open-Vocabulary Action Localization with Iterative Visual Prompting](https://mlsanity.com/paper/2408.17422/) (2024)
- [Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling](https://mlsanity.com/paper/2308.09946/) (2023)
- [Fine-grained Dynamic Network for Generic Event Boundary Detection](https://mlsanity.com/paper/2407.04274/) (2024)
- [Actor-centered Representations for Action Localization in Streaming Videos](https://mlsanity.com/paper/2104.14131/) (2021)
- [Unsupervised Audio-Visual Lecture Segmentation](https://mlsanity.com/paper/2210.16644/) (2022)
- [2by2: Weakly-Supervised Learning for Global Action Segmentation](https://mlsanity.com/paper/2412.12829/) (2024)
- [TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering](https://mlsanity.com/paper/2303.05166/) (2023)
- [Self-supervised Multi-actor Social Activity Understanding in Streaming Videos](https://mlsanity.com/paper/2406.14472/) (2024)
- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [Bringing Online Egocentric Action Recognition into the wild](https://mlsanity.com/paper/2211.03004/) (2022)

## Abstract (arXiv)

Temporal segmentation of long videos is an important problem, that has largely been tackled through supervised learning, often requiring large amounts of annotated training data. In this paper, we tackle the problem of self-supervised temporal segmentation of long videos that alleviate the need for any supervision. We introduce a self-supervised, predictive learning framework that draws inspiration from cognitive psychology to segment long, visually complex videos into individual, stable segments that share the same semantics. We also introduce a new adaptive learning paradigm that helps reduce the effect of catastrophic forgetting in recurrent neural networks. Extensive experiments on three publicly available datasets - Breakfast Actions, 50 Salads, and INRIA Instructional Videos datasets show the efficacy of the proposed approach. We show that the proposed approach is able to outperform weakly-supervised and other unsupervised learning approaches by up to 24% and have competitive performance compared to fully supervised approaches. We also show that the proposed approach is able to learn highly discriminative features that help improve action recognition when used in a representation learning paradigm.