# Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

arXiv: https://arxiv.org/abs/2011.09846 · HTML: https://mlsanity.com/paper/2011.09846/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2020.

## Built on by (most cited first, 30 of 34)

- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives](https://mlsanity.com/paper/2107.11317/) (2021)
- [Sign Language Translation from Instructional Videos](https://mlsanity.com/paper/2304.06371/) (2023)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting](https://mlsanity.com/paper/2210.05404/) (2022)
- [AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation](https://mlsanity.com/paper/2107.10685/) (2021)
- [Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation](https://mlsanity.com/paper/2207.03714/) (2022)
- [Giving a Hand to Diffusion Models: a Two-Stage Approach to Improving Conditional Human Image Generation](https://mlsanity.com/paper/2403.10731/) (2024)
- [Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production](https://mlsanity.com/paper/2112.05277/) (2021)
- [An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation](https://mlsanity.com/paper/2305.17714/) (2023)
- [Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization](https://mlsanity.com/paper/2506.15980/) (2025)
- [All You Need In Sign Language Production](https://mlsanity.com/paper/2201.01609/) (2022)
- [Sign Stitching: A Novel Approach to Sign Language Production](https://mlsanity.com/paper/2405.07663/) (2024)
- [Pose-Guided Fine-Grained Sign Language Video Generation](https://mlsanity.com/paper/2409.16709/) (2024)
- [iSign: A Benchmark for Indian Sign Language Processing](https://mlsanity.com/paper/2407.05404/) (2024)
- [SignSplat: Rendering Sign Language via Gaussian Splatting](https://mlsanity.com/paper/2505.02108/) (2025)
- [SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space](https://mlsanity.com/paper/2504.16315/) (2025)
- [SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Work](https://mlsanity.com/paper/2508.06951/) (2025)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Select and Reorder: A Novel Approach for Neural Sign Language Production](https://mlsanity.com/paper/2404.11532/) (2024)
- [Using Sign Language Production as Data Augmentation to enhance Sign Language Translation](https://mlsanity.com/paper/2506.09643/) (2025)
- [Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting](https://mlsanity.com/paper/2209.01470/) (2022)
- [Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Loss](https://mlsanity.com/paper/2409.09149/) (2024)
- [Stable Signer: Hierarchical Sign Language Generative Model](https://mlsanity.com/paper/2512.04048/) (2025)
- [M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production](https://mlsanity.com/paper/2603.23617/) (2026)
- [Topic Detection in Continuous Sign Language Videos](https://mlsanity.com/paper/2209.02402/) (2022)
- [Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods](https://mlsanity.com/paper/2308.12419/) (2023)
- [Generation and Detection of Sign Language Deepfakes - A Linguistic and Visual Analysis](https://mlsanity.com/paper/2404.01438/) (2024)
- [Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder](https://mlsanity.com/paper/2405.10423/) (2024)
- [Empowering Sign Language Communication: Integrating Sentiment and Semantics for Facial Expression Synthesis](https://mlsanity.com/paper/2408.15159/) (2024)

## Abstract (arXiv)

To be truly understandable and accepted by Deaf communities, an automatic Sign Language Production (SLP) system must generate a photo-realistic signer. Prior approaches based on graphical avatars have proven unpopular, whereas recent neural SLP works that produce skeleton pose sequences have been shown to be not understandable to Deaf viewers. In this paper, we propose SignGAN, the first SLP model to produce photo-realistic continuous sign language videos directly from spoken language. We employ a transformer architecture with a Mixture Density Network (MDN) formulation to handle the translation from spoken language to skeletal pose. A pose-conditioned human synthesis model is then introduced to generate a photo-realistic sign language video from the skeletal pose sequence. This allows the photo-realistic production of sign videos directly translated from written text. We further propose a novel keypoint-based loss function, which significantly improves the quality of synthesized hand images, operating in the keypoint space to avoid issues caused by motion blur. In addition, we introduce a method for controllable video generation, enabling training on large, diverse sign language datasets and providing the ability to control the signer appearance at inference. Using a dataset of eight different sign language interpreters extracted from broadcast footage, we show that SignGAN significantly outperforms all baseline methods for quantitative metrics and human perceptual studies.