# Linguistically Motivated Sign Language Segmentation

arXiv: https://arxiv.org/abs/2310.13960 · HTML: https://mlsanity.com/paper/2310.13960/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 21 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Real-Time Sign Language Detection using Human Pose Estimation](https://mlsanity.com/paper/2008.04637/) (2020)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [SHAS: Approaching optimal Segmentation for End-to-End Speech Translation](https://mlsanity.com/paper/2202.04774/) (2022)
- [Sign language segmentation with temporal convolutional networks](https://mlsanity.com/paper/2011.12986/) (2020)
- [Accelerating Recurrent Neural Networks for Gravitational Wave Experiments](https://mlsanity.com/paper/2106.14089/) (2021)
- [Scaling up sign spotting through sign language dictionaries](https://mlsanity.com/paper/2205.04152/) (2022)
- [Sign Segmentation with Changepoint-Modulated Pseudo-Labelling](https://mlsanity.com/paper/2104.13817/) (2021)
- [Towards the extraction of robust sign embeddings for low resource sign language recognition](https://mlsanity.com/paper/2306.17558/) (2023)
- [On the Importance of Signer Overlap for Sign Language Detection](https://mlsanity.com/paper/2303.10782/) (2023)

## Built on by (most cited first, 11 of 11)

- [Hands-On: Segmenting Individual Signs from Continuous Sequences](https://mlsanity.com/paper/2504.08593/) (2025)
- [Sign Stitching: A Novel Approach to Sign Language Production](https://mlsanity.com/paper/2405.07663/) (2024)
- [SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning](https://mlsanity.com/paper/2603.10446/) (2026)
- [IlluSign: Illustrating Sign Language Videos by Leveraging the Attention Mechanism](https://mlsanity.com/paper/2504.10822/) (2025)
- [Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios](https://mlsanity.com/paper/2508.18183/) (2025)
- [An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface](https://mlsanity.com/paper/2408.09311/) (2024)
- [iLSU-T: an Open Dataset for Uruguayan Sign Language Translation](https://mlsanity.com/paper/2507.21104/) (2025)
- [The TUB Sign Language Corpus Collection](https://mlsanity.com/paper/2508.05374/) (2025)
- [MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition](https://mlsanity.com/paper/2511.19907/) (2025)
- [Toward Signing Activity Projection in Sign Language Interaction](https://mlsanity.com/paper/2606.09424/) (2026)
- [SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval](https://mlsanity.com/paper/2609.03695/) (2026)

## Abstract (arXiv)

Sign language segmentation is a crucial task in sign language processing systems. It enables downstream tasks such as sign recognition, transcription, and machine translation. In this work, we consider two kinds of segmentation: segmentation into individual signs and segmentation into phrases, larger units comprising several signs. We propose a novel approach to jointly model these two tasks. Our method is motivated by linguistic cues observed in sign language corpora. We replace the predominant IO tagging scheme with BIO tagging to account for continuous signing. Given that prosody plays a significant role in phrase boundaries, we explore the use of optical flow features. We also provide an extensive analysis of hand shapes and 3D hand normalization. We find that introducing BIO tagging is necessary to model sign boundaries. Explicitly encoding prosody by optical flow improves segmentation in shallow models, but its contribution is negligible in deeper models. Careful tuning of the decoding algorithm atop the models further improves the segmentation quality. We demonstrate that our final models generalize to out-of-domain video content in a different signed language, even under a zero-shot setting. We observe that including optical flow and 3D hand normalization enhances the robustness of the model in this context.