DiffSign: AI-Assisted Generation of Customizable Sign Language Videos With Enhanced Realism
On the map: Diffusion models → Sign language production
Abstract (arXiv)
The proliferation of several streaming services in recent years has now made it possible for a diverse audience across the world to view the same media content, such as movies or TV shows. While translation and dubbing services are being added to make content accessible to the local audience, the support for making content accessible to people with different abilities, such as the Deaf and Hard of Hearing (DHH) community, is still lagging. Our goal is to make media content more accessible to the DHH community by generating sign language videos with synthetic signers that are realistic and expressive. Using the same signer for a given media content that is viewed globally may have limited appeal. Hence, our approach combines parametric modeling and generative modeling to generate realistic-looking synthetic signers and customize their appearance based on user preferences. We first retarget human sign language poses to 3D sign language avatars by optimizing a parametric model. The high-fidelity poses from the rendered avatars are then used to condition the poses of synthetic signers generated using a diffusion-based generative model. The appearance of the synthetic signer is controlled by an image prompt supplied through a visual adapter. Our results show that the sign language videos generated using our approach have better temporal consistency and realism than signing videos generated by a diffusion model conditioned only on text prompts. We also support multimodal prompts to allow users to further customize the appearance of the signer to accommodate diversity (e.g. skin tone, gender). Our approach is also useful for signer anonymization.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sign language production, one of 14 directions in Diffusion models. Builds on 12 of its 18 references that are in the map. First posted on arXiv 5 Dec 2024.
Builds on
12 of 18 references.
References · 12
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
- 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
- 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
- 2017Are GANs Created Equal? A Large-Scale Studycited by 267
- 2022Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Productioncited by 61
- 2022Changing the Representation: Examining Language Representation for Neural Sign Language Productioncited by 15
- 2022Neural Sign Reenactor: Deep Photorealistic Sign Language Retargetingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SignRefine: Adapting Foundational Video Models for Sign Language Generation
- 2025Stable Signer: Hierarchical Sign Language Generative Model
- 2025Diverse Signer Avatars with Manual and Non-Manual Feature Modelling for Sign Language Production
- 2025Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization
- 2026SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance
- 2023SignDiff: Diffusion Model for American Sign Language Production
- 2025IlluSign: Illustrating Sign Language Videos by Leveraging the Attention Mechanism
- 2025Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Production