# SLTUNET: A Simple Unified Model for Sign Language Translation

arXiv: https://arxiv.org/abs/2305.01778 · HTML: https://mlsanity.com/paper/2305.01778/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 69 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [One Model To Learn Them All](https://mlsanity.com/paper/1706.05137/) (2017)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022)
- [Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production](https://mlsanity.com/paper/2203.15354/) (2022)
- [Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation](https://mlsanity.com/paper/2102.05766/) (2021)
- [Boosting Continuous Sign Language Recognition via Cross Modality Augmentation](https://mlsanity.com/paper/2010.05264/) (2020)

## Built on by (most cited first, 30 of 35)

- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2405.04164/) (2024)
- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval](https://mlsanity.com/paper/2408.09949/) (2024)
- [Cross-modality Data Augmentation for End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.11096/) (2023)
- [Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2411.16789/) (2024)
- [Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation](https://mlsanity.com/paper/2505.15438/) (2025)
- [Hands-On: Segmenting Individual Signs from Continuous Sequences](https://mlsanity.com/paper/2504.08593/) (2025)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Scaling Sign Language Translation](https://mlsanity.com/paper/2407.11855/) (2024)
- [CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation](https://mlsanity.com/paper/2404.11111/) (2024)
- [Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2507.23575/) (2025)
- [EvSign: Sign Language Recognition and Translation with Streaming Events](https://mlsanity.com/paper/2407.12593/) (2024)
- [Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment](https://mlsanity.com/paper/2512.08040/) (2025)
- [A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production](https://mlsanity.com/paper/2407.02854/) (2024)
- [Unsupervised Sign Language Translation and Generation](https://mlsanity.com/paper/2402.07726/) (2024)
- [SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space](https://mlsanity.com/paper/2504.16315/) (2025)
- [GLoT: A Novel Gated-Logarithmic Transformer for Efficient Sign Language Translation](https://mlsanity.com/paper/2502.12223/) (2025)
- [RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2512.07273/) (2025)
- [DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model](https://mlsanity.com/paper/2411.17248/) (2024)
- [ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation](https://mlsanity.com/paper/2504.11942/) (2025)
- [SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translation](https://mlsanity.com/paper/2507.09266/) (2025)
- [SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation](https://mlsanity.com/paper/2509.00030/) (2025)
- [SignDATA: Data Pipeline for Sign Language Translation](https://mlsanity.com/paper/2604.20357/) (2026)
- [SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation](https://mlsanity.com/paper/2606.28626/) (2026)
- [SSL-SSAW: Self-Supervised Learning with Sigmoid Self-Attention Weighting for Question-Based Sign Language Translation](https://mlsanity.com/paper/2509.14036/) (2025)
- [HATL: Hierarchical Adaptive-Transfer Learning Framework for Sign Language Machine Translation](https://mlsanity.com/paper/2603.19260/) (2026)

## Abstract (arXiv)

Despite recent successes with neural models for sign language translation (SLT), translation quality still lags behind spoken languages because of the data scarcity and modality gap between sign video and text. To address both problems, we investigate strategies for cross-modality representation sharing for SLT. We propose SLTUNET, a simple unified neural model designed to support multiple SLTrelated tasks jointly, such as sign-to-gloss, gloss-to-text and sign-to-text translation. Jointly modeling different tasks endows SLTUNET with the capability to explore the cross-task relatedness that could help narrow the modality gap. In addition, this allows us to leverage the knowledge from external resources, such as abundant parallel data used for spoken-language machine translation (MT). We show in experiments that SLTUNET achieves competitive and even state-of-the-art performance on PHOENIX-2014T and CSL-Daily when augmented with MT data and equipped with a set of optimization techniques. We further use the DGS Corpus for end-to-end SLT for the first time. It covers broader domains with a significantly larger vocabulary, which is more challenging and which we consider to allow for a more realistic assessment of the current state of SLT than the former two. Still, SLTUNET obtains improved results on the DGS Corpus. Code is available at https://github.com/bzhangGo/sltunet.