# SMC-NCA: Semantic-guided Multi-level Contrast for Semi-supervised Temporal Action Segmentation

arXiv: https://arxiv.org/abs/2312.12347 · HTML: https://mlsanity.com/paper/2312.12347/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 91 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 30 of 59)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [Humble Teachers Teach Better Students for Semi-Supervised Object Detection](https://mlsanity.com/paper/2106.10456/) (2021)
- [Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples](https://mlsanity.com/paper/2104.13963/) (2021)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Autonomous Vehicles that Interact with Pedestrians: A Survey of Theory and Practice](https://mlsanity.com/paper/1805.11773/) (2018)
- [Unsupervised learning of action classes with continuous temporal embedding](https://mlsanity.com/paper/1904.04189/) (2019)

## Built on by (most cited first, 1 of 1)

- [Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding](https://mlsanity.com/paper/2510.10682/) (2025)

## Abstract (arXiv)

Semi-supervised temporal action segmentation (SS-TA) aims to perform frame-wise classification in long untrimmed videos, where only a fraction of videos in the training set have labels. Recent studies have shown the potential of contrastive learning in unsupervised representation learning using unlabelled data. However, learning the representation of each frame by unsupervised contrastive learning for action segmentation remains an open and challenging problem. In this paper, we propose a novel Semantic-guided Multi-level Contrast scheme with a Neighbourhood-Consistency-Aware unit (SMC-NCA) to extract strong frame-wise representations for SS-TAS. Specifically, for representation learning, SMC is first used to explore intra- and inter-information variations in a unified and contrastive way, based on action-specific semantic information and temporal information highlighting relations between actions. Then, the NCA module, which is responsible for enforcing spatial consistency between neighbourhoods centered at different frames to alleviate over-segmentation issues, works alongside SMC for semi-supervised learning (SSL). Our SMC outperforms the other state-of-the-art methods on three benchmarks, offering improvements of up to 17.8% and 12.6% in terms of Edit distance and accuracy, respectively. Additionally, the NCA unit results in significantly better segmentation performance in the presence of only 5% labelled videos. We also demonstrate the generalizability and effectiveness of the proposed method on our Parkinson Disease's Mouse Behaviour (PDMB) dataset. Code is available at https://github.com/FeixiangZhou/SMC-NCA.