# SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation

arXiv: https://arxiv.org/abs/2003.14266 · HTML: https://mlsanity.com/paper/2003.14266/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2020.

## Built on by (most cited first, 30 of 39)

- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Global2Local: Efficient Structure Search for Video Action Segmentation](https://mlsanity.com/paper/2101.00910/) (2021)
- [Action Shuffle Alternating Learning for Unsupervised Action Segmentation](https://mlsanity.com/paper/2104.02116/) (2021)
- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)
- [Hierarchical Vector Quantization for Unsupervised Action Segmentation](https://mlsanity.com/paper/2412.17640/) (2024)
- [Timestamp-Supervised Action Segmentation with Graph Convolutional Networks](https://mlsanity.com/paper/2206.15031/) (2022)
- [Leveraging Action Affinity and Continuity for Semi-supervised Temporal Action Segmentation](https://mlsanity.com/paper/2207.08653/) (2022)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [Temporal Action Segmentation with High-level Complex Activity Labels](https://mlsanity.com/paper/2108.06706/) (2021)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [Cross-Enhancement Transformer for Action Segmentation](https://mlsanity.com/paper/2205.09445/) (2022)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Leveraging triplet loss for unsupervised action segmentation](https://mlsanity.com/paper/2304.06403/) (2023)
- [Anchor-Constrained Viterbi for Set-Supervised Action Segmentation](https://mlsanity.com/paper/2104.02113/) (2021)
- [Pedestrian Attribute Recognition in Video Surveillance Scenarios Based on View-attribute Attention Localization](https://mlsanity.com/paper/2106.06485/) (2021)
- [CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation](https://mlsanity.com/paper/2109.15170/) (2021)
- [Robust Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2210.06501/) (2022)
- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [Coherent Temporal Synthesis for Incremental Action Segmentation](https://mlsanity.com/paper/2403.06102/) (2024)
- [Timestamp-Supervised Action Segmentation from the Perspective of Clustering](https://mlsanity.com/paper/2212.11694/) (2022)
- [Condensing Action Segmentation Datasets via Generative Network Inversion](https://mlsanity.com/paper/2503.14112/) (2025)
- [HOI-aware Adaptive Network for Weakly-supervised Action Segmentation](https://mlsanity.com/paper/2604.26227/) (2026)
- [Unsupervised Discovery of Actions in Instructional Videos](https://mlsanity.com/paper/2106.14733/) (2021)
- [TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering](https://mlsanity.com/paper/2303.05166/) (2023)
- [A Generalized & Robust Framework For Timestamp Supervision in Temporal Action Segmentation](https://mlsanity.com/paper/2207.10137/) (2022)

## Abstract (arXiv)

Temporal action segmentation is a topic of increasing interest, however, annotating each frame in a video is cumbersome and costly. Weakly supervised approaches therefore aim at learning temporal action segmentation from videos that are only weakly labeled. In this work, we assume that for each training video only the list of actions is given that occur in the video, but not when, how often, and in which order they occur. In order to address this task, we propose an approach that can be trained end-to-end on such data. The approach divides the video into smaller temporal regions and predicts for each region the action label and its length. In addition, the network estimates the action labels for each frame. By measuring how consistent the frame-wise predictions are with respect to the temporal regions and the annotated action labels, the network learns to divide a video into class-consistent regions. We evaluate our approach on three datasets where the approach achieves state-of-the-art results.