# Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting

arXiv: https://arxiv.org/abs/2210.05404 · HTML: https://mlsanity.com/paper/2210.05404/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 63 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective](https://mlsanity.com/paper/1908.08597/) (2019)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Neural Sign Language Translation based on Human Keypoint Estimation](https://mlsanity.com/paper/1811.11436/) (2018)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [Adversarial Training for Multi-Channel Sign Language Production](https://mlsanity.com/paper/2008.12405/) (2020)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)
- [Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video](https://mlsanity.com/paper/2011.09846/) (2020)
- [Real-Time Sign Language Detection using Human Pose Estimation](https://mlsanity.com/paper/2008.04637/) (2020)
- [Distributionally Robust Multilingual Machine Translation](https://mlsanity.com/paper/2109.04020/) (2021)
- [Changing the Representation: Examining Language Representation for Neural Sign Language Production](https://mlsanity.com/paper/2210.06312/) (2022)

## Built on by (most cited first, 18 of 18)

- [Ham2Pose: Animating Sign Language Notation into Pose Sequences](https://mlsanity.com/paper/2211.13613/) (2022)
- [SignCLIP: Connecting Text and Sign Language by Contrastive Learning](https://mlsanity.com/paper/2407.01264/) (2024)
- [JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing](https://mlsanity.com/paper/2311.10174/) (2023)
- [A Data-Driven Representation for Sign Language Production](https://mlsanity.com/paper/2404.11499/) (2024)
- [iSign: A Benchmark for Indian Sign Language Processing](https://mlsanity.com/paper/2407.05404/) (2024)
- [Meaningful Pose-Based Sign Language Evaluation](https://mlsanity.com/paper/2510.07453/) (2025)
- [Disentangle and Regularize: Sign Language Production with Articulator-Based Disentanglement and Channel-Aware Regularization](https://mlsanity.com/paper/2504.06610/) (2025)
- [Select and Reorder: A Novel Approach for Neural Sign Language Production](https://mlsanity.com/paper/2404.11532/) (2024)
- [SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation](https://mlsanity.com/paper/2406.06648/) (2024)
- [signwriting-evaluation: Effective Sign Language Evaluation via SignWriting](https://mlsanity.com/paper/2410.13668/) (2024)
- [Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios](https://mlsanity.com/paper/2508.18183/) (2025)
- [SignBank+: Preparing a Multilingual Sign Language Dataset for Machine Translation Using Large Language Models](https://mlsanity.com/paper/2309.11566/) (2023)
- [sign.mt: Real-Time Multilingual Sign Language Translation Application](https://mlsanity.com/paper/2310.05064/) (2023)
- [Evaluation of Pose Estimation Systems for Sign Language Translation](https://mlsanity.com/paper/2604.24609/) (2026)
- [KAN Text to Vision? The Exploration of Kolmogorov-Arnold Networks for Multi-Scale Sequence-Based Pose Animation from Sign Language Notation](https://mlsanity.com/paper/2605.09572/) (2026)
- [Direct Translation between Sign Languages](https://mlsanity.com/paper/2605.20588/) (2026)
- [A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language](https://mlsanity.com/paper/2608.10588/) (2026)
- [A Hybrid Kinematic GCN and Elastic Transformer for Efficient Indian Sign Language Recognition](https://mlsanity.com/paper/2610.14680/) (2026)

## Abstract (arXiv)

This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup--translating from American Sign Language to (American) English--our method achieves over 30 BLEU, while in two multilingual setups--translating in both directions between spoken languages and signed languages--we achieve over 20 BLEU. We find that common MT techniques used to improve spoken language translation similarly affect the performance of sign language translation. These findings validate our use of an intermediate text representation for signed languages to include them in natural language processing research.