# Generation and Detection of Sign Language Deepfakes - A Linguistic and Visual Analysis

arXiv: https://arxiv.org/abs/2404.01438 · HTML: https://mlsanity.com/paper/2404.01438/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 20 of 20)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [DreamVideo: Composing Your Dream Videos with Customized Subject and Motion](https://mlsanity.com/paper/2312.04433/) (2023)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising](https://mlsanity.com/paper/2305.18264/) (2023)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Combining EfficientNet and Vision Transformers for Video Deepfake Detection](https://mlsanity.com/paper/2107.02612/) (2021)
- [Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video](https://mlsanity.com/paper/2011.09846/) (2020)
- [3DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translation](https://mlsanity.com/paper/1905.00307/) (2019)
- [FakeBuster: A DeepFakes Detection Tool for Video Conferencing Scenarios](https://mlsanity.com/paper/2101.03321/) (2021)

## Built on by (most cited first, 1 of 1)

- [1M-Deepfakes Detection Challenge](https://mlsanity.com/paper/2409.06991/) (2024)

## Abstract (arXiv)

This research explores the positive application of deepfake technology for upper body generation, specifically sign language for the Deaf and Hard of Hearing (DHoH) community. Given the complexity of sign language and the scarcity of experts, the generated videos are vetted by a sign language expert for accuracy. We construct a reliable deepfake dataset, evaluating its technical and visual credibility using computer vision and natural language processing models. The dataset, consisting of over 1200 videos featuring both seen and unseen individuals, is also used to detect deepfake videos targeting vulnerable individuals. Expert annotations confirm that the generated videos are comparable to real sign language content. Linguistic analysis, using textual similarity scores and interpreter evaluations, shows that the interpretation of generated videos is at least 90% similar to authentic sign language. Visual analysis demonstrates that convincingly realistic deepfakes can be produced, even for new subjects. Using a pose/style transfer model, we pay close attention to detail, ensuring hand movements are accurate and align with the driving video. We also apply machine learning algorithms to establish a baseline for deepfake detection on this dataset, contributing to the detection of fraudulent sign language videos.