mlsanityData to 7 Oct 2026

SignDiff: Diffusion Model for American Sign Language Production

arXiv 2308.16082 ↗First posted 30 Aug 2023cs.CVcited by 25

On the map: Diffusion models → Sign language production

Abstract (arXiv)

In this paper, we propose a dual-condition diffusion pre-training model named SignDiff that can generate human sign language speakers from a skeleton pose. SignDiff has a novel Frame Reinforcement Network called FR-Net, similar to dense human pose estimation work, which enhances the correspondence between text lexical symbols and sign language dense pose frames, reduces the occurrence of multiple fingers in the diffusion model. In addition, we propose a new method for American Sign Language Production (ASLP), which can generate ASL skeletal pose videos from text input, integrating two new improved modules and a new loss function to improve the accuracy and quality of sign language skeletal posture and enhance the ability of the model to train on large-scale data. We propose the first baseline for ASL production and report the scores of 17.19 and 12.85 on BLEU-4 on the How2Sign dev/test sets. We evaluated our model on the previous mainstream dataset PHOENIX14T, and the experiments achieved the SOTA results. In addition, our image quality far exceeds all previous results by 10 percentage points in terms of SSIM.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Sign language production, one of 14 directions in Diffusion models. Builds on 49 of its 79 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

49 of 79 references.

References · 49

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  8. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  9. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  10. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  11. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  12. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
37 more
  1. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  2. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  3. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  4. 2018Video-to-Video Synthesiscited by 388
  5. 2016Generating Videos with Scene Dynamicscited by 376
  6. 2018Everybody Dance Nowcited by 313
  7. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  8. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  9. 2017Pose Guided Person Image Generationcited by 251
  10. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  11. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  12. 2019Few-shot Video-to-Video Synthesiscited by 196
  13. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  14. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  15. 2017Deformable GANs for Pose-based Human Image Generationcited by 166
  16. 2020Controllable Person Image Synthesis with Attribute-Decomposed GANcited by 150
  17. 2019Progressive Pose Attention Transfer for Person Image Generationcited by 138
  18. 2020Progressive Transformers for End-to-End Sign Language Productioncited by 119
  19. 2018Synthesizing Images of Humans in Unseen Posescited by 96
  20. 2020XingGAN for Person Image Generationcited by 81
  21. 2020World-Consistent Video-to-Video Synthesiscited by 69
  22. 2018Neural Sign Language Translation based on Human Keypoint Estimationcited by 65
  23. 2020CONFIG: Controllable Neural Face Image Generationcited by 61
  24. 2022Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Productioncited by 61
  25. 2020Adversarial Training for Multi-Channel Sign Language Productioncited by 57
  26. 2020Quantitative Survey of the State of the Art in Sign Language Recognitioncited by 50
  27. 2021Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networkscited by 50
  28. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  29. 2019Dance Dance Generation: Motion Transfer for Internet Videoscited by 28
  30. 2018GestureGAN for Hand Gesture-to-Gesture Translation in the Wildcited by 27
  31. 2021Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionarycited by 27
  32. 2021AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisationcited by 16
  33. 2019GAC-GAN: A General Method for Appearance-Controllable Human Video Motion Transfercited by 14
  34. 2020Can Everybody Sign Now? Exploring Sign Language Video Generation from 2D Posescited by 12
  35. 2021Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Productioncited by 11
  36. 2019Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Mapscited by 7
  37. 2020MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesiscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 12.

25 papers.

Sort:

By area:

  1. 2024T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Textcited by 30
  2. 2024A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatarscited by 18
  3. 2024Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Productioncited by 16
  4. 2024Pose-Guided Fine-Grained Sign Language Video Generationcited by 8
  5. 2025SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Workcited by 4
  6. 2024Modelling the Distribution of Human Motion for Sign Language Assessmentcited by 3
  7. 2025SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Spacecited by 3
  8. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  9. 2025Stable Signer: Hierarchical Sign Language Generative Modelcited by 2
  10. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  11. 2026M3T: Discrete Multi-Modal Motion Tokens for Sign Language Productioncited by 2
  12. 2025StreamFlow: Theory, Algorithm, and Implementation for High-Efficiency Rectified Flow Generationcited by 1
13 more
  1. 2026Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Productioncited by 1
  2. 2024Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Modelcited by 0
  3. 2025Human-Centered Editable Speech-to-Sign-Language Generation via Streaming Conformer-Transformer and Resampling Hookcited by 0
  4. 2025Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learningcited by 0
  5. 2025EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generationcited by 0
  6. 2026RAC: Rectified Flow Auto Codercited by 0
  7. 2026SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languagescited by 0
  8. 2026Physics-Informed Diffusion for Biomechanically Plausible 3D Sign Language Generationcited by 0
  9. 2026SignRefine: Adapting Foundational Video Models for Sign Language Generationcited by 0
  10. 2026SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generationcited by 0
  11. 2026PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmarkcited by 0
  12. 2026RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robotscited by 0
  13. 2026Machine Translation for Sign Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Text-Driven Diffusion Model for Sign Language Production
  2. 2025Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production
  3. 2025Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Production
  4. 2023Neural Sign Actors: A diffusion model for 3D sign language production from text
  5. 2022G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model
  6. 2026SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks
  7. 2026The Impact of VAE Design on Latent Pose Representations for Diffusion-based Sign Language Production
  8. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model