# Faster Diffusion Action Segmentation

arXiv: https://arxiv.org/abs/2408.02024 · HTML: https://mlsanity.com/paper/2408.02024/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 4 Aug 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [An end-to-end generative framework for video segmentation and recognition](https://mlsanity.com/paper/1509.01947/) (2015)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [How Much Temporal Long-Term Context is Needed for Action Segmentation?](https://mlsanity.com/paper/2308.11358/) (2023)

## Built on by (most cited first, 2 of 2)

- [Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation](https://mlsanity.com/paper/2604.15173/) (2026)
- [Inconsistency-aware Multimodal Schr"odinger Bridge for Deepfake Localization](https://mlsanity.com/paper/2605.23113/) (2026)

## Abstract (arXiv)

Temporal Action Segmentation (TAS) is an essential task in video analysis, aiming to segment and classify continuous frames into distinct action segments. However, the ambiguous boundaries between actions pose a significant challenge for high-precision segmentation. Recent advances in diffusion models have demonstrated substantial success in TAS tasks due to their stable training process and high-quality generation capabilities. However, the heavy sampling steps required by diffusion models pose a substantial computational burden, limiting their practicality in real-time applications. Additionally, most related works utilize Transformer-based encoder architectures. Although these architectures excel at capturing long-range dependencies, they incur high computational costs and face feature-smoothing issues when processing long video sequences. To address these challenges, we propose EffiDiffAct, an efficient and high-performance TAS algorithm. Specifically, we develop a lightweight temporal feature encoder that reduces computational overhead and mitigates the rank collapse phenomenon associated with traditional self-attention mechanisms. Furthermore, we introduce an adaptive skip strategy that allows for dynamic adjustment of timestep lengths based on computed similarity metrics during inference, thereby further enhancing computational efficiency. Comprehensive experiments on the 50Salads, Breakfast, and GTEA datasets demonstrated the effectiveness of the proposed algorithm.