# Progressive Transformers for End-to-End Sign Language Production

arXiv: https://arxiv.org/abs/2004.14874 · HTML: https://mlsanity.com/paper/2004.14874/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 120)

- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production](https://mlsanity.com/paper/2203.15354/) (2022)
- [Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks](https://mlsanity.com/paper/2103.06982/) (2021)
- [Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives](https://mlsanity.com/paper/2107.11317/) (2021)
- [Neural Sign Actors: A diffusion model for 3D sign language production from text](https://mlsanity.com/paper/2312.02702/) (2023)
- [Content4All Open Research Sign Language Translation Datasets](https://mlsanity.com/paper/2105.02351/) (2021)
- [T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text](https://mlsanity.com/paper/2406.07119/) (2024)
- [SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark](https://mlsanity.com/paper/2310.20436/) (2023)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [Skeletor: Skeletal Transformers for Robust Body-Pose Estimation](https://mlsanity.com/paper/2104.11712/) (2021)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [SignDiff: Diffusion Model for American Sign Language Production](https://mlsanity.com/paper/2308.16082/) (2023)
- [Ham2Pose: Animating Sign Language Notation into Pose Sequences](https://mlsanity.com/paper/2211.13613/) (2022)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting](https://mlsanity.com/paper/2210.05404/) (2022)
- [A Data-Driven Representation for Sign Language Production](https://mlsanity.com/paper/2404.11499/) (2024)
- [AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation](https://mlsanity.com/paper/2107.10685/) (2021)
- [Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation](https://mlsanity.com/paper/2411.16810/) (2024)
- [Changing the Representation: Examining Language Representation for Neural Sign Language Production](https://mlsanity.com/paper/2210.06312/) (2022)
- [Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation](https://mlsanity.com/paper/2207.03714/) (2022)
- [SignAvatar: Sign Language 3D Motion Reconstruction and Generation](https://mlsanity.com/paper/2405.07974/) (2024)
- [Towards Zero-shot Sign Language Recognition](https://mlsanity.com/paper/2201.05914/) (2022)
- [Sign Language Production with Latent Motion Transformer](https://mlsanity.com/paper/2312.12917/) (2023)
- [FixMyPose: Pose Correctional Captioning and Retrieval](https://mlsanity.com/paper/2104.01703/) (2021)
- [Including Facial Expressions in Contextual Embeddings for Sign Language Generation](https://mlsanity.com/paper/2202.05383/) (2022)
- [Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production](https://mlsanity.com/paper/2112.05277/) (2021)
- [Modeling Intensification for Sign Language Generation: A Computational Approach](https://mlsanity.com/paper/2203.09679/) (2022)

## Abstract (arXiv)

The goal of automatic Sign Language Production (SLP) is to translate spoken language to a continuous stream of sign language video at a level comparable to a human translator. If this was achievable, then it would revolutionise Deaf hearing communications. Previous work on predominantly isolated SLP has shown the need for architectures that are better suited to the continuous domain of full sign sequences. In this paper, we propose Progressive Transformers, a novel architecture that can translate from discrete spoken language sentences to continuous 3D skeleton pose outputs representing sign language. We present two model configurations, an end-to-end network that produces sign direct from text and a stacked network that utilises a gloss intermediary. Our transformer network architecture introduces a counter that enables continuous sequence generation at training and inference. We also provide several data augmentation processes to overcome the problem of drift and improve the performance of SLP models. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging RWTH-PHOENIX-Weather-2014T(PHOENIX14T) dataset and setting baselines for future research.