# An end-to-end generative framework for video segmentation and recognition

arXiv: https://arxiv.org/abs/1509.01947 · HTML: https://mlsanity.com/paper/1509.01947/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2015.

## Built on by (most cited first, 30 of 61)

- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Global2Local: Efficient Structure Search for Video Action Segmentation](https://mlsanity.com/paper/2101.00910/) (2021)
- [How Much Temporal Long-Term Context is Needed for Action Segmentation?](https://mlsanity.com/paper/2308.11358/) (2023)
- [Action Shuffle Alternating Learning for Unsupervised Action Segmentation](https://mlsanity.com/paper/2104.02116/) (2021)
- [Coarse to Fine Multi-Resolution Temporal Convolutional Network](https://mlsanity.com/paper/2105.10859/) (2021)
- [Learning to Ground Instructional Articles in Videos through Narrations](https://mlsanity.com/paper/2306.03802/) (2023)
- [Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities](https://mlsanity.com/paper/2105.00067/) (2021)
- [Action Segmentation with Mixed Temporal Domain Adaptation](https://mlsanity.com/paper/2104.07461/) (2021)
- [Hierarchical Vector Quantization for Unsupervised Action Segmentation](https://mlsanity.com/paper/2412.17640/) (2024)
- [Timestamp-Supervised Action Segmentation with Graph Convolutional Networks](https://mlsanity.com/paper/2206.15031/) (2022)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [FIFA: Fast Inference Approximation for Action Segmentation](https://mlsanity.com/paper/2108.03894/) (2021)
- [Temporal Action Segmentation with High-level Complex Activity Labels](https://mlsanity.com/paper/2108.06706/) (2021)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [Cross-Enhancement Transformer for Action Segmentation](https://mlsanity.com/paper/2205.09445/) (2022)
- [FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation](https://mlsanity.com/paper/2402.00293/) (2024)
- [Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos](https://mlsanity.com/paper/2110.05697/) (2021)
- [OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation](https://mlsanity.com/paper/2309.06276/) (2023)
- [Step Differences in Instructional Video](https://mlsanity.com/paper/2404.16222/) (2024)
- [Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks](https://mlsanity.com/paper/2103.15449/) (2021)
- [Sign Segmentation with Changepoint-Modulated Pseudo-Labelling](https://mlsanity.com/paper/2104.13817/) (2021)
- [Do we really need temporal convolutions in action segmentation?](https://mlsanity.com/paper/2205.13425/) (2022)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Leveraging triplet loss for unsupervised action segmentation](https://mlsanity.com/paper/2304.06403/) (2023)

## Abstract (arXiv)

We describe an end-to-end generative approach for the segmentation and recognition of human activities. In this approach, a visual representation based on reduced Fisher Vectors is combined with a structured temporal model for recognition. We show that the statistical properties of Fisher Vectors make them an especially suitable front-end for generative models such as Gaussian mixtures. The system is evaluated for both the recognition of complex activities as well as their parsing into action units. Using a variety of video datasets ranging from human cooking activities to animal behaviors, our experiments demonstrate that the resulting architecture outperforms state-of-the-art approaches for larger datasets, i.e. when sufficient amount of data is available for training structured generative models.