# Robust Action Segmentation from Timestamp Supervision

arXiv: https://arxiv.org/abs/2210.06501 · HTML: https://mlsanity.com/paper/2210.06501/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment](https://mlsanity.com/paper/1803.10699/) (2018)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling](https://mlsanity.com/paper/1703.08132/) (2017)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [SF-Net: Single-Frame Supervision for Temporal Action Localization](https://mlsanity.com/paper/2003.06845/) (2020)
- [An end-to-end generative framework for video segmentation and recognition](https://mlsanity.com/paper/1509.01947/) (2015)
- [Weakly Supervised Energy-Based Learning for Action Segmentation](https://mlsanity.com/paper/1909.13155/) (2019)
- [Weakly Supervised Action Labeling in Videos Under Ordering Constraints](https://mlsanity.com/paper/1407.1208/) (2014)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Weakly supervised learning of actions from transcripts](https://mlsanity.com/paper/1610.02237/) (2016)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [Global2Local: Efficient Structure Search for Video Action Segmentation](https://mlsanity.com/paper/2101.00910/) (2021)
- [Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2108.05029/) (2021)
- [SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation](https://mlsanity.com/paper/2003.14266/) (2020)
- [A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation](https://mlsanity.com/paper/1906.01028/) (2019)
- [Set-Constrained Viterbi for Set-Supervised Action Segmentation](https://mlsanity.com/paper/2002.11925/) (2020)
- [Action Recognition from Single Timestamp Supervision in Untrimmed Videos](https://mlsanity.com/paper/1904.04689/) (2019)
- [Fast Weakly Supervised Action Segmentation Using Mutual Consistency](https://mlsanity.com/paper/1904.03116/) (2019)
- [FIFA: Fast Inference Approximation for Action Segmentation](https://mlsanity.com/paper/2108.03894/) (2021)
- [Anchor-Constrained Viterbi for Set-Supervised Action Segmentation](https://mlsanity.com/paper/2104.02113/) (2021)

## Built on by (most cited first, 6 of 6)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [Coherent Temporal Synthesis for Incremental Action Segmentation](https://mlsanity.com/paper/2403.06102/) (2024)
- [Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actions](https://mlsanity.com/paper/2508.05529/) (2025)
- [Hierarchical Action Learning for Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2602.24275/) (2026)
- [Robust Surgical Phase Recognition From Annotation Efficient Supervision](https://mlsanity.com/paper/2406.18481/) (2024)

## Abstract (arXiv)

Action segmentation is the task of predicting an action label for each frame of an untrimmed video. As obtaining annotations to train an approach for action segmentation in a fully supervised way is expensive, various approaches have been proposed to train action segmentation models using different forms of weak supervision, e.g., action transcripts, action sets, or more recently timestamps. Timestamp supervision is a promising type of weak supervision as obtaining one timestamp per action is less expensive than annotating all frames, but it provides more information than other forms of weak supervision. However, previous works assume that every action instance is annotated with a timestamp, which is a restrictive assumption since it assumes that annotators do not miss any action. In this work, we relax this restrictive assumption and take missing annotations for some action instances into account. We show that our approach is more robust to missing annotations compared to other approaches and various baselines.