# Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives

arXiv: https://arxiv.org/abs/2107.11317 · HTML: https://mlsanity.com/paper/2107.11317/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 70 references that are in the map.
- First posted on arXiv 23 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Conditional Computation in Neural Networks for faster models](https://mlsanity.com/paper/1511.06297/) (2015)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [One Model To Learn Them All](https://mlsanity.com/paper/1706.05137/) (2017)
- [Mixture Models for Diverse Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/1902.07816/) (2019)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Neural Sign Language Translation based on Human Keypoint Estimation](https://mlsanity.com/paper/1811.11436/) (2018)
- [Adversarial Training for Multi-Channel Sign Language Production](https://mlsanity.com/paper/2008.12405/) (2020)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks](https://mlsanity.com/paper/2103.06982/) (2021)
- [Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts](https://mlsanity.com/paper/2002.04013/) (2020)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)
- [Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video](https://mlsanity.com/paper/2011.09846/) (2020)
- [Content4All Open Research Sign Language Translation Datasets](https://mlsanity.com/paper/2105.02351/) (2021)
- [Mixture Content Selection for Diverse Sequence Generation](https://mlsanity.com/paper/1909.01953/) (2019)
- [A Mixture of h-1 Heads is Better than h Heads](https://mlsanity.com/paper/2005.06537/) (2020)
- [Neural Sign Language Translation by Learning Tokenization](https://mlsanity.com/paper/2002.00479/) (2020)
- [AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation](https://mlsanity.com/paper/2107.10685/) (2021)
- [Sequence to Sequence Mixture Model for Diverse Machine Translation](https://mlsanity.com/paper/1810.07391/) (2018)
- [Mixtures of Experts Models](https://mlsanity.com/paper/1806.08200/) (2018)

## Built on by (most cited first, 30 of 50)

- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production](https://mlsanity.com/paper/2203.15354/) (2022)
- [Neural Sign Actors: A diffusion model for 3D sign language production from text](https://mlsanity.com/paper/2312.02702/) (2023)
- [Sign Language Translation from Instructional Videos](https://mlsanity.com/paper/2304.06371/) (2023)
- [G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model](https://mlsanity.com/paper/2208.09141/) (2022)
- [T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text](https://mlsanity.com/paper/2406.07119/) (2024)
- [SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark](https://mlsanity.com/paper/2310.20436/) (2023)
- [SignDiff: Diffusion Model for American Sign Language Production](https://mlsanity.com/paper/2308.16082/) (2023)
- [Ham2Pose: Animating Sign Language Notation into Pose Sequences](https://mlsanity.com/paper/2211.13613/) (2022)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [A Data-Driven Representation for Sign Language Production](https://mlsanity.com/paper/2404.11499/) (2024)
- [Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation](https://mlsanity.com/paper/2411.16810/) (2024)
- [Changing the Representation: Examining Language Representation for Neural Sign Language Production](https://mlsanity.com/paper/2210.06312/) (2022)
- [Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation](https://mlsanity.com/paper/2207.03714/) (2022)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [Including Facial Expressions in Contextual Embeddings for Sign Language Generation](https://mlsanity.com/paper/2202.05383/) (2022)
- [T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences](https://mlsanity.com/paper/2406.00636/) (2024)
- [Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production](https://mlsanity.com/paper/2112.05277/) (2021)
- [Modeling Intensification for Sign Language Generation: A Computational Approach](https://mlsanity.com/paper/2203.09679/) (2022)
- [Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representations](https://mlsanity.com/paper/2309.12179/) (2023)
- [Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization](https://mlsanity.com/paper/2506.15980/) (2025)
- [Sign Stitching: A Novel Approach to Sign Language Production](https://mlsanity.com/paper/2405.07663/) (2024)
- [Pose-Guided Fine-Grained Sign Language Video Generation](https://mlsanity.com/paper/2409.16709/) (2024)
- [Transcribing Natural Languages for The Deaf via Neural Editing Programs](https://mlsanity.com/paper/2112.09600/) (2021)
- [Meaningful Pose-Based Sign Language Evaluation](https://mlsanity.com/paper/2510.07453/) (2025)
- [A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production](https://mlsanity.com/paper/2407.02854/) (2024)
- [Text-Driven 3D Hand Motion Generation from Sign Language Data](https://mlsanity.com/paper/2508.15902/) (2025)
- [Non-Autoregressive Sign Language Production via Knowledge Distillation](https://mlsanity.com/paper/2208.06183/) (2022)
- [MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production](https://mlsanity.com/paper/2407.12842/) (2024)
- [SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space](https://mlsanity.com/paper/2504.16315/) (2025)

## Abstract (arXiv)

It is common practice to represent spoken languages at their phonetic level. However, for sign languages, this implies breaking motion into its constituent motion primitives. Avatar based Sign Language Production (SLP) has traditionally done just this, building up animation from sequences of hand motions, shapes and facial expressions. However, more recent deep learning based solutions to SLP have tackled the problem using a single network that estimates the full skeletal structure. We propose splitting the SLP task into two distinct jointly-trained sub-tasks. The first translation sub-task translates from spoken language to a latent sign language representation, with gloss supervision. Subsequently, the animation sub-task aims to produce expressive sign language sequences that closely resemble the learnt spatio-temporal representation. Using a progressive transformer for the translation sub-task, we propose a novel Mixture of Motion Primitives (MoMP) architecture for sign language animation. A set of distinct motion primitives are learnt during training, that can be temporally combined at inference to animate continuous sign language sequences. We evaluate on the challenging RWTH-PHOENIX-Weather-2014T(PHOENIX14T) dataset, presenting extensive ablation studies and showing that MoMP outperforms baselines in user evaluations. We achieve state-of-the-art back translation performance with an 11% improvement over competing results. Importantly, and for the first time, we showcase stronger performance for a full translation pipeline going from spoken language to sign, than from gloss to sign.