# An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

arXiv: https://arxiv.org/abs/2305.17714 · HTML: https://mlsanity.com/paper/2305.17714/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 77 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video](https://mlsanity.com/paper/2011.09846/) (2020)
- [Data Augmentation for Sign Language Gloss Translation](https://mlsanity.com/paper/2105.07476/) (2021)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [Ham2Pose: Animating Sign Language Notation into Pose Sequences](https://mlsanity.com/paper/2211.13613/) (2022)
- [AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation](https://mlsanity.com/paper/2107.10685/) (2021)
- [Can Everybody Sign Now? Exploring Sign Language Video Generation from 2D Poses](https://mlsanity.com/paper/2012.10941/) (2020)
- [Sockeye 3: Fast Neural Machine Translation with PyTorch](https://mlsanity.com/paper/2207.05851/) (2022)

## Built on by (most cited first, 9 of 9)

- [Meaningful Pose-Based Sign Language Evaluation](https://mlsanity.com/paper/2510.07453/) (2025)
- [Explanatory Summarization with Discourse-Driven Planning](https://mlsanity.com/paper/2504.19339/) (2025)
- [Pose-Based Sign Language Appearance Transfer](https://mlsanity.com/paper/2410.13675/) (2024)
- [Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder](https://mlsanity.com/paper/2405.10423/) (2024)
- [MATE: LLM-Powered Multi-Agent Translation Environment for Accessibility Applications](https://mlsanity.com/paper/2506.19502/) (2025)
- [Introducing A Bangla Sentence - Gloss Pair Dataset for Bangla Sign Language Translation and Research](https://mlsanity.com/paper/2511.08507/) (2025)
- [sign.mt: Real-Time Multilingual Sign Language Translation Application](https://mlsanity.com/paper/2310.05064/) (2023)
- [An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface](https://mlsanity.com/paper/2408.09311/) (2024)
- [Evaluation of Pose Estimation Systems for Sign Language Translation](https://mlsanity.com/paper/2604.24609/) (2026)

## Abstract (arXiv)

Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.