# Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learning

arXiv: https://arxiv.org/abs/2512.04219 · HTML: https://mlsanity.com/paper/2512.04219/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 70 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Fast and Accurate Online Video Object Segmentation via Tracking Parts](https://mlsanity.com/paper/1806.02323/) (2018)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation](https://mlsanity.com/paper/2003.14266/) (2020)
- [A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation](https://mlsanity.com/paper/1906.01028/) (2019)
- [How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks](https://mlsanity.com/paper/2407.03475/) (2024)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval](https://mlsanity.com/paper/2204.00486/) (2022)

## Abstract (arXiv)

Humans naturally perceive continuous experience as a hierarchy of temporally nested events, fine-grained actions embedded within coarser routines. Replicating this structure in computer vision requires models that can segment video not just retrospectively, but predictively and hierarchically. We introduce PARSE, a unified framework that learns multiscale event structure directly from streaming video without supervision. PARSE organizes perception into a hierarchy of recurrent predictors, each operating at its own temporal granularity: lower layers model short-term dynamics while higher layers integrate longer-term context through attention-based feedback. Event boundaries emerge naturally as transient peaks in prediction error, yielding temporally coherent, nested partonomies that mirror the containment relations observed in human event perception. Evaluated across three benchmarks, Breakfast Actions, 50 Salads, and Assembly 101, PARSE achieves state-of-the-art performance among streaming methods and rivals offline baselines in both temporal alignment (H-GEBD) and structural consistency (TED, hF1). The results demonstrate that predictive learning under uncertainty provides a scalable path toward human-like temporal abstraction and compositional event understanding.