# Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos

arXiv: https://arxiv.org/abs/2609.31148 · HTML: https://mlsanity.com/paper/2609.31148/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation](https://mlsanity.com/paper/2003.14266/) (2020)
- [Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2404.01518/) (2024)
- [Watch, read and lookup: learning to spot signs from multiple supervisors](https://mlsanity.com/paper/2010.04002/) (2020)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [Improving Continuous Sign Language Recognition with Cross-Lingual Signs](https://mlsanity.com/paper/2308.10809/) (2023)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [Zero-shot keyword spotting for visual speech recognition in-the-wild](https://mlsanity.com/paper/1807.08469/) (2018)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [Cross-Enhancement Transformer for Action Segmentation](https://mlsanity.com/paper/2205.09445/) (2022)
- [Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos](https://mlsanity.com/paper/2209.05653/) (2022)
- [Sign Language Translation with Sentence Embedding Supervision](https://mlsanity.com/paper/2510.19367/) (2025)
- [Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustment](https://mlsanity.com/paper/2408.09919/) (2024)

## Abstract (arXiv)

Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply when applied to long, continuous sign language videos. To bridge this gap, we focus on a challenging and realistic setting: Visual-only Sentence-level Sign Language Segmentation (Vis-SSLS), which aims to partition continuous sign language videos into non-overlapping sentence-level segments without any caption assistance, serving as a crucial prerequisite for downstream recognition and translation tasks. However, sentence transitions in sign language are often smooth and visually ambiguous, lacking explicit pauses or posture resets. As a result, static frame representations may fail to capture the subtle temporal changes that indicate sentence boundaries. To address this challenge, we propose SignShift, a difference-aware segmentation framework that explicitly models frame-to-frame feature variation as semantic cues for sentence boundary detection. First, to model the feature variation, we design a Temporal Difference Module, which incorporates full-frame, facial, and hand cues, and employs inter-frame differencing to learn multi-scale temporal variations that capture both fine-grained local kinematics and global semantic transitions. Second, to mitigate over- and under-segmentation issues, we design a Segment Count Prediction module, which predicts the number of sentences to guide boundary selection. Extensive experiments on benchmark datasets demonstrate that SignShift substantially outperforms existing methods, validating its effectiveness.