# MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation

arXiv: https://arxiv.org/abs/2006.09220 · HTML: https://mlsanity.com/paper/2006.09220/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 19 of 19)

- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Global2Local: Efficient Structure Search for Video Action Segmentation](https://mlsanity.com/paper/2101.00910/) (2021)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [Fingerspelling Detection in American Sign Language](https://mlsanity.com/paper/2104.01291/) (2021)
- [Using Hand Pose Estimation To Automate Open Surgery Training Feedback](https://mlsanity.com/paper/2211.07021/) (2022)
- [Depth Super-Resolution from Explicit and Implicit High-Frequency Features](https://mlsanity.com/paper/2303.09307/) (2023)
- [Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips](https://mlsanity.com/paper/2112.01038/) (2021)
- [Prompt-enhanced Hierarchical Transformer Elevating Cardiopulmonary Resuscitation Instruction via Temporal Action Segmentation](https://mlsanity.com/paper/2308.16552/) (2023)
- [Tencent AVS: A Holistic Ads Video Dataset for Multi-modal Scene Segmentation](https://mlsanity.com/paper/2212.04700/) (2022)
- [Few-Shot Radar Signal Recognition through Self-Supervised Learning and Radio Frequency Domain Adaptation](https://mlsanity.com/paper/2501.03461/) (2025)
- [Multi-Slice Net: A novel light weight framework for COVID-19 Diagnosis](https://mlsanity.com/paper/2108.03786/) (2021)
- [Temporal2Seq: A Unified Framework for Temporal Video Understanding Tasks](https://mlsanity.com/paper/2409.18478/) (2024)
- [End-to-End Action Segmentation Transformer](https://mlsanity.com/paper/2503.06316/) (2025)
- [Multi-Stage Boundary-Aware Transformer Network for Action Segmentation in Untrimmed Surgical Videos](https://mlsanity.com/paper/2504.18756/) (2025)
- [Enhancing Video-Based Robot Failure Detection Using Task Knowledge](https://mlsanity.com/paper/2508.18705/) (2025)
- [A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking](https://mlsanity.com/paper/2511.17805/) (2025)
- [MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation](https://mlsanity.com/paper/2608.29891/) (2026)

## Abstract (arXiv)

With the success of deep learning in classifying short trimmed videos, more attention has been focused on temporally segmenting and classifying activities in long untrimmed videos. State-of-the-art approaches for action segmentation utilize several layers of temporal convolution and temporal pooling. Despite the capabilities of these approaches in capturing temporal dependencies, their predictions suffer from over-segmentation errors. In this paper, we propose a multi-stage architecture for the temporal action segmentation task that overcomes the limitations of the previous approaches. The first stage generates an initial prediction that is refined by the next ones. In each stage we stack several layers of dilated temporal convolutions covering a large receptive field with few parameters. While this architecture already performs well, lower layers still suffer from a small receptive field. To address this limitation, we propose a dual dilated layer that combines both large and small receptive fields. We further decouple the design of the first stage from the refining stages to address the different requirements of these stages. Extensive evaluation shows the effectiveness of the proposed model in capturing long-range dependencies and recognizing action segments. Our models achieve state-of-the-art results on three datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.