# Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder

arXiv: https://arxiv.org/abs/2405.10423 · HTML: https://mlsanity.com/paper/2405.10423/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [VITON: An Image-based Virtual Try-on Network](https://mlsanity.com/paper/1711.08447/) (2017)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production](https://mlsanity.com/paper/2203.15354/) (2022)
- [Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video](https://mlsanity.com/paper/2011.09846/) (2020)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [Multi-Channel Attention Selection GANs for Guided Image-to-Image Translation](https://mlsanity.com/paper/2002.01048/) (2020)
- [AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation](https://mlsanity.com/paper/2107.10685/) (2021)

## Built on by (most cited first, 1 of 1)

- [Diverse Signer Avatars with Manual and Non-Manual Feature Modelling for Sign Language Production](https://mlsanity.com/paper/2508.15988/) (2025)

## Abstract (arXiv)

This paper addresses the problem of diversity-aware sign language production, where we want to give an image (or sequence) of a signer and produce another image with the same pose but different attributes (textite.g. gender, skin color). To this end, we extend the variational inference paradigm to include information about the pose and the conditioning of the attributes. This formulation improves the quality of the synthesised images. The generator framework is presented as a UNet architecture to ensure spatial preservation of the input pose, and we include the visual features from the variational inference to maintain control over appearance and style. We generate each body part with a separate decoder. This architecture allows the generator to deliver better overall results. Experiments on the SMILE II dataset show that the proposed model performs quantitatively better than state-of-the-art baselines regarding diversity, per-pixel image quality, and pose estimation. Quantitatively, it faithfully reproduces non-manual features for signers.