# Towards Online Continuous Sign Language Recognition and Translation

arXiv: https://arxiv.org/abs/2401.05336 · HTML: https://mlsanity.com/paper/2401.05336/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Whole-Body Human Pose Estimation in the Wild](https://mlsanity.com/paper/2007.11858/) (2020)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Fully Convolutional Networks for Continuous Sign Language Recognition](https://mlsanity.com/paper/2007.12402/) (2020)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)
- [CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment](https://mlsanity.com/paper/2303.05725/) (2023)
- [Global-local Enhancement Network for NMFs-aware Sign Language Recognition](https://mlsanity.com/paper/2008.10428/) (2020)
- [Watch, read and lookup: learning to spot signs from multiple supervisors](https://mlsanity.com/paper/2010.04002/) (2020)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [Transformer-based Online CTC/attention End-to-End Speech Recognition Architecture](https://mlsanity.com/paper/2001.08290/) (2020)
- [Improving Continuous Sign Language Recognition with Cross-Lingual Signs](https://mlsanity.com/paper/2308.10809/) (2023)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Scaling Up Online Speech Recognition Using ConvNets](https://mlsanity.com/paper/2001.09727/) (2020)
- [An Online Attention-based Model for Speech Recognition](https://mlsanity.com/paper/1811.05247/) (2018)

## Built on by (most cited first, 14 of 14)

- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2601.06843/) (2026)
- [A Hong Kong Sign Language Corpus Collected from Sign-interpreted TV News](https://mlsanity.com/paper/2405.00980/) (2024)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Large Sign Language Models: Toward 3D American Sign Language Translation](https://mlsanity.com/paper/2511.08535/) (2025)
- [Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Production](https://mlsanity.com/paper/2606.21234/) (2026)
- [Stack Transformer Based Spatial-Temporal Attention Model for Dynamic Sign Language and Fingerspelling Recognition](https://mlsanity.com/paper/2503.16855/) (2025)
- [Towards Multi-View Sign Language Understanding: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2604.18184/) (2026)
- [Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching](https://mlsanity.com/paper/2606.11925/) (2026)
- [SignRR: Retrieve and Refine Real Motion for Sign Language Production](https://mlsanity.com/paper/2608.28568/) (2026)
- [SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation](https://mlsanity.com/paper/2609.06296/) (2026)
- [ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation](https://mlsanity.com/paper/2610.03022/) (2026)

## Abstract (arXiv)

Research on continuous sign language recognition (CSLR) is essential to bridge the communication gap between deaf and hearing individuals. Numerous previous studies have trained their models using the connectionist temporal classification (CTC) loss. During inference, these CTC-based models generally require the entire sign video as input to make predictions, a process known as offline recognition, which suffers from high latency and substantial memory usage. In this work, we take the first step towards online CSLR. Our approach consists of three phases: 1) developing a sign dictionary; 2) training an isolated sign language recognition model on the dictionary; and 3) employing a sliding window approach on the input sign sequence, feeding each sign clip to the optimized model for online recognition. Additionally, our online recognition model can be extended to support online translation by integrating a gloss-to-text network and can enhance the performance of any offline model. With these extensions, our online approach achieves new state-of-the-art performance on three popular benchmarks across various task settings. Code and models are available at https://github.com/FangyunWei/SLRT.