# Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities

arXiv: https://arxiv.org/abs/2105.00067 · HTML: https://mlsanity.com/paper/2105.00067/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [Timeception for Complex Action Recognition](https://mlsanity.com/paper/1812.01289/) (2018)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Unsupervised learning of action classes with continuous temporal embedding](https://mlsanity.com/paper/1904.04189/) (2019)
- [Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment](https://mlsanity.com/paper/1803.10699/) (2018)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling](https://mlsanity.com/paper/1703.08132/) (2017)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [An end-to-end generative framework for video segmentation and recognition](https://mlsanity.com/paper/1509.01947/) (2015)
- [Unsupervised Learning and Segmentation of Complex Activities from Video](https://mlsanity.com/paper/1803.09490/) (2018)
- [VideoGraph: Recognizing Minutes-Long Human Activities in Videos](https://mlsanity.com/paper/1905.05143/) (2019)
- [Weakly Supervised Energy-Based Learning for Action Segmentation](https://mlsanity.com/paper/1909.13155/) (2019)
- [Weakly supervised learning of actions from transcripts](https://mlsanity.com/paper/1610.02237/) (2016)
- [Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences](https://mlsanity.com/paper/2001.11122/) (2020)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation](https://mlsanity.com/paper/1906.01028/) (2019)
- [What's Cookin'? Interpreting Cooking Videos using Text, Speech and Vision](https://mlsanity.com/paper/1503.01558/) (2015)
- [Unsupervised Semantic Parsing of Video Collections](https://mlsanity.com/paper/1506.08438/) (2015)

## Built on by (most cited first, 26 of 26)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2404.01518/) (2024)
- [Hierarchical Vector Quantization for Unsupervised Action Segmentation](https://mlsanity.com/paper/2412.17640/) (2024)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation](https://mlsanity.com/paper/2309.06276/) (2023)
- [Joint Self-Supervised Video Alignment and Action Segmentation](https://mlsanity.com/paper/2503.16832/) (2025)
- [CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2507.03539/) (2025)
- [Open Vocabulary Multi-Label Video Classification](https://mlsanity.com/paper/2407.09073/) (2024)
- [A Culturally-diverse Multilingual Multimodal Video Benchmark & Model](https://mlsanity.com/paper/2506.07032/) (2025)
- [Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos](https://mlsanity.com/paper/2510.11204/) (2025)
- [Towards Tokenized Human Dynamics Representation](https://mlsanity.com/paper/2111.11433/) (2021)
- [2by2: Weakly-Supervised Learning for Global Action Segmentation](https://mlsanity.com/paper/2412.12829/) (2024)
- [Learning Video-independent Eye Contact Segmentation from In-the-Wild Videos](https://mlsanity.com/paper/2210.02033/) (2022)
- [Improving action segmentation via explicit similarity measurement](https://mlsanity.com/paper/2502.10713/) (2025)
- [Biomechanical-phase based Temporal Segmentation in Sports Videos: a Demonstration on Javelin-Throw](https://mlsanity.com/paper/2509.24606/) (2025)
- [Privacy Beyond Pixels: Latent Anonymization for Privacy-Preserving Video Understanding](https://mlsanity.com/paper/2511.08666/) (2025)
- [Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation](https://mlsanity.com/paper/2602.21406/) (2026)
- [Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approach](https://mlsanity.com/paper/2607.05271/) (2026)
- ["Knights": First Place Submission for VIPriors21 Action Recognition Challenge at ICCV 2021](https://mlsanity.com/paper/2110.07758/) (2021)
- [StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales](https://mlsanity.com/paper/2510.16209/) (2025)
- [Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics](https://mlsanity.com/paper/2602.24138/) (2026)
- [Deep kernel video approximation for unsupervised action segmentation](https://mlsanity.com/paper/2604.21572/) (2026)
- [D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2608.05877/) (2026)
- [See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning](https://mlsanity.com/paper/2608.29611/) (2026)
- [FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2608.29980/) (2026)

## Abstract (arXiv)

Action recognition and detection in the context of long untrimmed video sequences has seen an increased attention from the research community. However, annotation of complex activities is usually time consuming and challenging in practice. Therefore, recent works started to tackle the problem of unsupervised learning of sub-actions in complex activities. This paper proposes a novel approach for unsupervised sub-action learning in complex activities. The proposed method maps both visual and temporal representations to a latent space where the sub-actions are learnt discriminatively in an end-to-end fashion. To this end, we propose to learn sub-actions as latent concepts and a novel discriminative latent concept learning (DLCL) module aids in learning sub-actions. The proposed DLCL module lends on the idea of latent concepts to learn compact representations in the latent embedding space in an unsupervised way. The result is a set of latent vectors that can be interpreted as cluster centers in the embedding space. The latent space itself is formed by a joint visual and temporal embedding capturing the visual similarity and temporal ordering of the data. Our joint learning with discriminative latent concept module is novel which eliminates the need for explicit clustering. We validate our approach on three benchmark datasets and show that the proposed combination of visual-temporal embedding and discriminative latent concepts allow to learn robust action representations in an unsupervised setting.