Simultaneous Translation between Sign Languages
Not on the map. Why?
Abstract (arXiv)
Deaf and hard-of-hearing (DHH) signers cannot converse in real time across different sign languages today: existing sign-to-sign translation systems run offline, requiring the full source clip before any target sign is emitted. Live use cases - e.g. broadcast interpretation and two-way video calls - instead demand simultaneous output, while the source signer is still signing. We present, to our knowledge, the first simultaneous sign-to-sign (S2S) translation system, with two wait-k regimes: test-time wait-k inference applied directly to a full-sentence model, and a trained wait-k model via stochastic multi-path supervision. We further introduce ca-Stream-AL, a computation-aware latency metric for streaming output. Averaged across six S2S directions on both a smaller human-verified test set and a larger synthetic S2S corpus, our streaming system achieves a 38% ca-Stream-AL reduction while staying within a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop compared to the full-sentence baseline. A word-order case study probes how the streaming model handles word order mismatch between different sign languages - a consequence of simultaneous translation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
13 of 21 references.
References · 13
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
- 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
- 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
- 2020Progressive Transformers for End-to-End Sign Language Productioncited by 119
- 2016Can neural machine translation do simultaneous translation?cited by 85
- 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
- 2020Efficient Wait-k Models for Simultaneous Machine Translationcited by 59
- 2023Neural Sign Actors: A diffusion model for 3D sign language production from textcited by 48
- 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
- 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
- 2020Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Trainingcited by 10
Built on by
Not cited yet.