# Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation

arXiv: https://arxiv.org/abs/2609.08167 · HTML: https://mlsanity.com/paper/2609.08167/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 59 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos](https://mlsanity.com/paper/2203.14104/) (2022)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [Timestamp-Supervised Action Segmentation with Graph Convolutional Networks](https://mlsanity.com/paper/2206.15031/) (2022)
- [TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation](https://mlsanity.com/paper/1705.07818/) (2017)
- [Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting](https://mlsanity.com/paper/2410.00771/) (2024)
- [Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure](https://mlsanity.com/paper/2403.07292/) (2024)
- [Efficient Temporal Action Segmentation via Boundary-aware Query Voting](https://mlsanity.com/paper/2405.15995/) (2024)
- [Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning](https://mlsanity.com/paper/2206.11011/) (2022)
- [Timestamp-Supervised Action Segmentation from the Perspective of Clustering](https://mlsanity.com/paper/2212.11694/) (2022)
- [HOI-aware Adaptive Network for Weakly-supervised Action Segmentation](https://mlsanity.com/paper/2604.26227/) (2026)
- [Hierarchical Context Transformer for Multi-level Semantic Scene Understanding](https://mlsanity.com/paper/2502.15184/) (2025)
- [Turning to a Teacher for Timestamp Supervised Temporal Action Segmentation](https://mlsanity.com/paper/2207.00712/) (2022)

## Abstract (arXiv)

Timestamp-supervised action segmentation aims to segment and classify actions in untrimmed videos with a random frame annotated per action. Precisely localizing action boundaries from timestamp annotations is crucial for this setting, as it enables generating framewise pseudo-labels and applying the well-explored fully-supervised training. However, prevailing methods struggle with intrinsic uncertainty in boundary localization due to less discriminative features in action-transiting regions. This imprecise boundary estimation significantly reduces the stability and reliability of the generated pseudo-labels in ambiguous action-transiting regions, consequently resulting in performance deterioration of the trained segmentation models. In our paper, we introduce the boundary voting network that mitigates feature ambiguity by hierarchically propagating video-level global prior knowledge into local action-transiting regions. By generating key action representations as votes throughout the video and targeting action-transiting regions, all votes collaboratively contribute to action-transiting feature enhancement and boundary localization refinement. Extensive experiments demonstrate the effectiveness of our method on GTEA, 50Salads, and Breakfast datasets.