# Including Facial Expressions in Contextual Embeddings for Sign Language Generation

arXiv: https://arxiv.org/abs/2202.05383 · HTML: https://mlsanity.com/paper/2202.05383/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks](https://mlsanity.com/paper/2103.06982/) (2021)
- [Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives](https://mlsanity.com/paper/2107.11317/) (2021)

## Built on by (most cited first, 12 of 12)

- [Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation](https://mlsanity.com/paper/2411.16810/) (2024)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [Modeling Intensification for Sign Language Generation: A Computational Approach](https://mlsanity.com/paper/2203.09679/) (2022)
- [Linguistics-Vision Monotonic Consistent Network for Sign Language Production](https://mlsanity.com/paper/2412.16944/) (2024)
- [SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluation](https://mlsanity.com/paper/2509.03791/) (2025)
- [SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generation](https://mlsanity.com/paper/2506.11621/) (2025)
- [Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Production](https://mlsanity.com/paper/2509.10845/) (2025)
- [Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Production](https://mlsanity.com/paper/2606.21234/) (2026)
- [SignNet: Single Channel Sign Generation using Metric Embedded Learning](https://mlsanity.com/paper/2212.02848/) (2022)
- [Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community](https://mlsanity.com/paper/2404.01158/) (2024)
- [EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generation](https://mlsanity.com/paper/2511.22135/) (2025)
- [DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation](https://mlsanity.com/paper/2609.02796/) (2026)

## Abstract (arXiv)

State-of-the-art sign language generation frameworks lack expressivity and naturalness which is the result of only focusing manual signs, neglecting the affective, grammatical and semantic functions of facial expressions. The purpose of this work is to augment semantic representation of sign language through grounding facial expressions. We study the effect of modeling the relationship between text, gloss, and facial expressions on the performance of the sign generation systems. In particular, we propose a Dual Encoder Transformer able to generate manual signs as well as facial expressions by capturing the similarities and differences found in text and sign gloss annotation. We take into consideration the role of facial muscle activity to express intensities of manual signs by being the first to employ facial action units in sign language generation. We perform a series of experiments showing that our proposed model improves the quality of automatically generated sign language.