# Unsupervised Action Segmentation for Instructional Videos

arXiv: https://arxiv.org/abs/2106.03738 · HTML: https://mlsanity.com/paper/2106.03738/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [Unsupervised learning of action classes with continuous temporal embedding](https://mlsanity.com/paper/1904.04189/) (2019)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling](https://mlsanity.com/paper/1703.08132/) (2017)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [Unsupervised Learning and Segmentation of Complex Activities from Video](https://mlsanity.com/paper/1803.09490/) (2018)
- [Weakly supervised learning of actions from transcripts](https://mlsanity.com/paper/1610.02237/) (2016)
- [AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures](https://mlsanity.com/paper/1905.13209/) (2019)
- [Set-Constrained Viterbi for Set-Supervised Action Segmentation](https://mlsanity.com/paper/2002.11925/) (2020)

## Built on by (most cited first, 4 of 4)

- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learning](https://mlsanity.com/paper/2310.00608/) (2023)
- [Generating Illustrated Instructions](https://mlsanity.com/paper/2312.04552/) (2023)
- [What You Say Is What You Show: Visual Narration Detection in Instructional Videos](https://mlsanity.com/paper/2301.02307/) (2023)

## Abstract (arXiv)

In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos, which are rarely annotated with atomic actions. We present an unsupervised approach to learn atomic actions of structured human tasks from a variety of instructional videos based on a sequential stochastic autoregressive model for temporal segmentation of videos. This learns to represent and discover the sequential relationship between different atomic actions of the task, and which provides automatic and unsupervised self-labeling.