# Weakly Supervised Action Labeling in Videos Under Ordering Constraints

arXiv: https://arxiv.org/abs/1407.1208 · HTML: https://mlsanity.com/paper/1407.1208/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2014.

## Built on by (most cited first, 30 of 48)

- [DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video](https://mlsanity.com/paper/2202.00164/) (2022)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Temporal Alignment Networks for Long-term Video](https://mlsanity.com/paper/2204.02968/) (2022)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations](https://mlsanity.com/paper/2303.17839/) (2023)
- [Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos](https://mlsanity.com/paper/2203.11637/) (2022)
- [TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency](https://mlsanity.com/paper/2208.06773/) (2022)
- [Action Shuffle Alternating Learning for Unsupervised Action Segmentation](https://mlsanity.com/paper/2104.02116/) (2021)
- [Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2203.16800/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Learning to Ground Instructional Articles in Videos through Narrations](https://mlsanity.com/paper/2306.03802/) (2023)
- [Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2108.06524/) (2021)
- [Learning Discriminative Prototypes with Dynamic Time Warping](https://mlsanity.com/paper/2103.09458/) (2021)
- [Weakly-supervised segmentation of referring expressions](https://mlsanity.com/paper/2205.04725/) (2022)
- [FIFA: Fast Inference Approximation for Action Segmentation](https://mlsanity.com/paper/2108.03894/) (2021)
- [Temporal Action Segmentation with High-level Complex Activity Labels](https://mlsanity.com/paper/2108.06706/) (2021)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [Less is More: Surgical Phase Recognition from Timestamp Supervision](https://mlsanity.com/paper/2202.08199/) (2022)
- [United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videos](https://mlsanity.com/paper/2311.03550/) (2023)
- [Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment](https://mlsanity.com/paper/2403.19225/) (2024)
- [Causal Temporal Representation Learning with Nonstationary Sparse Transition](https://mlsanity.com/paper/2409.03142/) (2024)
- [Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation](https://mlsanity.com/paper/2505.15438/) (2025)
- [Timestamp-supervised Wearable-based Activity Segmentation and Recognition with Contrastive Learning and Order-Preserving Optimal Transport](https://mlsanity.com/paper/2310.09114/) (2023)
- [Multi-Sentence Grounding for Long-term Instructional Video](https://mlsanity.com/paper/2312.14055/) (2023)

## Abstract (arXiv)

We are given a set of video clips, each one annotated with an em ordered list of actions, such as "walk" then "sit" then "answer phone" extracted from, for example, the associated text script. We seek to temporally localize the individual actions in each clip as well as to learn a discriminative classifier for each action. We formulate the problem as a weakly supervised temporal assignment with ordering constraints. Each video clip is divided into small time intervals and each time interval of each video clip is assigned one action label, while respecting the order in which the action labels appear in the given annotations. We show that the action label assignment can be determined together with learning a classifier for each action in a discriminative manner. We evaluate the proposed model on a new and challenging dataset of 937 video clips with a total of 787720 frames containing sequences of 16 different actions from 69 Hollywood movies.