# Efficient Temporal Action Segmentation via Boundary-aware Query Voting

arXiv: https://arxiv.org/abs/2405.15995 · HTML: https://mlsanity.com/paper/2405.15995/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation](https://mlsanity.com/paper/2003.02824/) (2020)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Detection Transformer with Stable Matching](https://mlsanity.com/paper/2304.04742/) (2023)
- [Global2Local: Efficient Structure Search for Video Action Segmentation](https://mlsanity.com/paper/2101.00910/) (2021)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation](https://mlsanity.com/paper/1906.01028/) (2019)
- [How Much Temporal Long-Term Context is Needed for Action Segmentation?](https://mlsanity.com/paper/2308.11358/) (2023)
- [Temporal Relational Modeling with Self-Supervision for Action Segmentation](https://mlsanity.com/paper/2012.07508/) (2020)
- [Activity Grammars for Temporal Action Segmentation](https://mlsanity.com/paper/2312.04266/) (2023)
- [DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation](https://mlsanity.com/paper/2304.02110/) (2023)

## Built on by (most cited first, 10 of 10)

- [Text-Derived Relational Graph-Enhanced Network for Skeleton-Based Action Segmentation](https://mlsanity.com/paper/2503.15126/) (2025)
- [End-to-End Action Segmentation Transformer](https://mlsanity.com/paper/2503.06316/) (2025)
- [From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-training in Industrial Settings](https://mlsanity.com/paper/2511.21428/) (2025)
- [MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment](https://mlsanity.com/paper/2603.00878/) (2026)
- [HRTR: A Single-stage Transformer for Fine-grained Sub-second Action Segmentation in Stroke Rehabilitation](https://mlsanity.com/paper/2506.02472/) (2025)
- [LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation](https://mlsanity.com/paper/2603.24097/) (2026)
- [Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation](https://mlsanity.com/paper/2603.24134/) (2026)
- [Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation](https://mlsanity.com/paper/2604.15173/) (2026)
- [Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation](https://mlsanity.com/paper/2609.08167/) (2026)
- [Groupwise Selective State-Space Filtering for Accurate and Streaming Action Boundary Detection](https://mlsanity.com/paper/2609.33400/) (2026)

## Abstract (arXiv)

Although the performance of Temporal Action Segmentation (TAS) has improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource-intensive post-processing requirements. To improve the efficiency while keeping the performance, we present a novel perspective centered on per-segment classification. By harnessing the capabilities of Transformers, we tokenize each video segment as an instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary-aware Transformer network. It employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs a simple yet effective voting strategy to classify boundary-wise segments based on instance segmentation. Remarkably, as a single-stage approach, BaFormer significantly reduces the computational costs, utilizing only 6% of the running time compared to state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at https://github.com/peiyao-w/BaFormer.