# SimulSLT: End-to-End Simultaneous Sign Language Translation

arXiv: https://arxiv.org/abs/2112.04228 · HTML: https://mlsanity.com/paper/2112.04228/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Neural Sign Language Translation based on Human Keypoint Estimation](https://mlsanity.com/paper/1811.11436/) (2018)
- [Neural Sign Language Translation by Learning Tokenization](https://mlsanity.com/paper/2002.00479/) (2020)
- [FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire](https://mlsanity.com/paper/2008.02516/) (2020)

## Built on by (most cited first, 27 of 27)

- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Gloss Attention for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2307.07361/) (2023)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation](https://mlsanity.com/paper/2205.12523/) (2022)
- [Machine Translation from Signed to Spoken Languages: State of the Art and Challenges](https://mlsanity.com/paper/2202.03086/) (2022)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2411.16789/) (2024)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment](https://mlsanity.com/paper/2306.06410/) (2023)
- [TrainerAgent: Customizable and Efficient Model Training through LLM-Powered Multi-Agent System](https://mlsanity.com/paper/2311.06622/) (2023)
- [From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation](https://mlsanity.com/paper/2408.14825/) (2024)
- [A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production](https://mlsanity.com/paper/2407.02854/) (2024)
- [SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval](https://mlsanity.com/paper/2407.16394/) (2024)
- [DATE: Domain Adaptive Product Seeker for E-commerce](https://mlsanity.com/paper/2304.03669/) (2023)
- [Language Model is a Branch Predictor for Simultaneous Machine Translation](https://mlsanity.com/paper/2312.14488/) (2023)
- [GLoT: A Novel Gated-Logarithmic Transformer for Efficient Sign Language Translation](https://mlsanity.com/paper/2502.12223/) (2025)
- [DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model](https://mlsanity.com/paper/2411.17248/) (2024)
- [ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation](https://mlsanity.com/paper/2504.11942/) (2025)
- [What's the Point? Spatial Grammar & Index Resolution for Sign Language Processing](https://mlsanity.com/paper/2606.08056/) (2026)
- [SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation](https://mlsanity.com/paper/2606.28626/) (2026)
- [Co-creation for Sign Language Processing and Machine Translation](https://mlsanity.com/paper/2503.01553/) (2025)
- [Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks](https://mlsanity.com/paper/2609.03734/) (2026)
- [Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation](https://mlsanity.com/paper/2609.07965/) (2026)
- [Reading Less While Writing: A Closed-Form Bandwidth Dial for Streaming Multimodal Decoders](https://mlsanity.com/paper/2609.20845/) (2026)
- [OneSign: Unifying Sign Language Understanding Tasks with One Model](https://mlsanity.com/paper/2609.33090/) (2026)
- [Machine Translation for Sign Languages](https://mlsanity.com/paper/2610.00881/) (2026)
- [ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation](https://mlsanity.com/paper/2610.03022/) (2026)

## Abstract (arXiv)

Sign language translation as a kind of technology with profound social significance has attracted growing researchers' interest in recent years. However, the existing sign language translation methods need to read all the videos before starting the translation, which leads to a high inference latency and also limits their application in real-life scenarios. To solve this problem, we propose SimulSLT, the first end-to-end simultaneous sign language translation model, which can translate sign language videos into target text concurrently. SimulSLT is composed of a text decoder, a boundary predictor, and a masked encoder. We 1) use the wait-k strategy for simultaneous translation. 2) design a novel boundary predictor based on the integrate-and-fire module to output the gloss boundary, which is used to model the correspondence between the sign language video and the gloss. 3) propose an innovative re-encode method to help the model obtain more abundant contextual information, which allows the existing video features to interact fully. The experimental results conducted on the RWTH-PHOENIX-Weather 2014T dataset show that SimulSLT achieves BLEU scores that exceed the latest end-to-end non-simultaneous sign language translation model while maintaining low latency, which proves the effectiveness of our method.