# Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting

arXiv: https://arxiv.org/abs/2209.01470 · HTML: https://mlsanity.com/paper/2209.01470/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 68 references that are in the map.
- First posted on arXiv 3 Sep 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Neural Kinematic Networks for Unsupervised Motion Retargetting](https://mlsanity.com/paper/1804.05653/) (2018)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production](https://mlsanity.com/paper/2203.15354/) (2022)
- [Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2006.01423/) (2020)
- [Adversarial Training for Multi-Channel Sign Language Production](https://mlsanity.com/paper/2008.12405/) (2020)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [Learning Character-Agnostic Motion for Motion Retargeting in 2D](https://mlsanity.com/paper/1905.01680/) (2019)
- [TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting](https://mlsanity.com/paper/2003.14401/) (2020)
- [Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation](https://mlsanity.com/paper/2109.13318/) (2021)
- [Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video](https://mlsanity.com/paper/2011.09846/) (2020)
- [Deep Video-Based Performance Cloning](https://mlsanity.com/paper/1808.06847/) (2018)
- [Head2Head++: Deep Facial Attributes Re-Targeting](https://mlsanity.com/paper/2006.10199/) (2020)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation](https://mlsanity.com/paper/2107.10685/) (2021)
- [MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks](https://mlsanity.com/paper/2112.10082/) (2021)

## Built on by (most cited first, 2 of 2)

- [DiffSLVA: Harnessing Diffusion Models for Sign Language Video Anonymization](https://mlsanity.com/paper/2311.16060/) (2023)
- [DiffSign: AI-Assisted Generation of Customizable Sign Language Videos With Enhanced Realism](https://mlsanity.com/paper/2412.03878/) (2024)

## Abstract (arXiv)

In this paper, we introduce a neural rendering pipeline for transferring the facial expressions, head pose, and body movements of one person in a source video to another in a target video. We apply our method to the challenging case of Sign Language videos: given a source video of a sign language user, we can faithfully transfer the performed manual (e.g., handshape, palm orientation, movement, location) and non-manual (e.g., eye gaze, facial expressions, mouth patterns, head, and body movements) signs to a target video in a photo-realistic manner. Our method can be used for Sign Language Anonymization, Sign Language Production (synthesis module), as well as for reenacting other types of full body activities (dancing, acting performance, exercising, etc.). We conduct detailed qualitative and quantitative evaluations and comparisons, which demonstrate the particularly promising and realistic results that we obtain and the advantages of our method over existing approaches.