mlsanityData to 7 Oct 2026

Neural Sign Actors: A diffusion model for 3D sign language production from text

arXiv 2312.02702 ↗First posted 5 Dec 2023cs.CVcited by 48

On the map: Diffusion models → Sign language production

Abstract (arXiv)

Sign Languages (SL) serve as the primary mode of communication for the Deaf and Hard of Hearing communities. Deep learning methods for SL recognition and translation have achieved promising results. However, Sign Language Production (SLP) poses a challenge as the generated motions must be realistic and have precise semantic meaning. Most SLP methods rely on 2D data, which hinders their realism. In this work, a diffusion-based SLP model is trained on a curated large-scale dataset of 4D signing avatars and their corresponding text transcripts. The proposed method can generate dynamic sequences of 3D avatars from an unconstrained domain of discourse using a diffusion process formed on a novel and anatomically informed graph neural network defined on the SMPL-X body skeleton. Through quantitative and qualitative experiments, we show that the proposed method considerably outperforms previous methods of SLP. This work makes an important step towards realistic neural sign avatars, bridging the communication gap between Deaf and hearing communities.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Sign language production, one of 14 directions in Diffusion models. Builds on 35 of its 68 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

35 of 68 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  7. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  8. 2022Human Motion Diffusion Modelcited by 1,062
  9. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  10. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  11. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  12. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
23 more
  1. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  2. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  3. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  4. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  5. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  6. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  7. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  8. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  9. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 122
  10. 2020Progressive Transformers for End-to-End Sign Language Productioncited by 119
  11. 2018MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Languagecited by 118
  12. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  13. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 104
  14. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  15. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  16. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  17. 2022Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Productioncited by 61
  18. 2020Adversarial Training for Multi-Channel Sign Language Productioncited by 57
  19. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  20. 2021Adversarial Parametric Pose Priorcited by 38
  21. 2023Reconstructing Signing Avatars From Video Using Linguistic Priorscited by 12
  22. 2022GraphWalks: Efficient Shape Agnostic Geodesic Shortest Path Estimationcited by 8
  23. 2021Towards Automatic Speech to Sign Language Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 21.

48 papers.

Sort:

By area:

  1. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
  2. 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 62
  3. 2024T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Textcited by 30
  4. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  5. 2025Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenizationcited by 9
  6. 2024Pose-Guided Fine-Grained Sign Language Video Generationcited by 8
  7. 2024AnimateMe: 4D Facial Expressions via Diffusion Modelscited by 6
  8. 2024ShapeFusion: A 3D diffusion model for localized shape editingcited by 6
  9. 2025Meaningful Pose-Based Sign Language Evaluationcited by 6
  10. 2024A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Productioncited by 5
  11. 2025Disentangle and Regularize: Sign Language Production with Articulator-Based Disentanglement and Channel-Aware Regularizationcited by 5
  12. 2025Text-Driven 3D Hand Motion Generation from Sign Language Datacited by 5
36 more
  1. 2025SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Workcited by 4
  2. 2025ImHead: A Large-scale Implicit Morphable Model for Localized Head Modelingcited by 4
  3. 2025SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generationcited by 3
  4. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  5. 2024SignLLM: Sign Language Production Large Language Modelscited by 2
  6. 2025Stable Signer: Hierarchical Sign Language Generative Modelcited by 2
  7. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  8. 2026M3T: Discrete Multi-Modal Motion Tokens for Sign Language Productioncited by 2
  9. 2026HandX: Scaling Bimanual Motion and Interaction Generationcited by 2
  10. 2024Invariant Representation Guided Multimodal Sentiment Decoding with Sequential Variation Regularizationcited by 1
  11. 2025SIGNER: Temporally Grounded Sign Language Generation via Time-Resolved Conditioningcited by 1
  12. 2025Teach Me Sign: Stepwise Prompting LLM for Sign Language Productioncited by 1
  13. 2025Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Productioncited by 1
  14. 2025DexAvatar: 3D Sign Language Reconstruction with Hand and Body Pose Priorscited by 1
  15. 2026Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysiscited by 1
  16. 2026Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Productioncited by 1
  17. 2026BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Productioncited by 1
  18. 2024Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Modelcited by 0
  19. 2025Bringing Balance to Hand Shape Classification: Mitigating Data Imbalance Through Generative Modelscited by 0
  20. 2025Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learningcited by 0
  21. 2025Geometry-Aware Losses for Structure-Preserving Text-to-Sign Language Generationcited by 0
  22. 2025EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generationcited by 0
  23. 2026OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesiscited by 0
  24. 2026SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languagescited by 0
  25. 2026Tamaththul3D: High-Fidelity 3D Saudi Sign Language Avatars from Monocular Videocited by 0
  26. 2026Conditional Collapse in Sign Language Production: A Diagnostic and a Scaling Argumentcited by 0
  27. 2026The Impact of VAE Design on Latent Pose Representations for Diffusion-based Sign Language Productioncited by 0
  28. 2026Physics-Informed Diffusion for Biomechanically Plausible 3D Sign Language Generationcited by 0
  29. 2026From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigationcited by 0
  30. 2026Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Productioncited by 0
  31. 2026DiscoSign: Discourse-Aware Text to Sign Language Gloss Translationcited by 0
  32. 2026SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generationcited by 0
  33. 2026PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmarkcited by 0
  34. 2026RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robotscited by 0
  35. 2026Naturalness-guided Manifold Flow Matching for Sign Language Productioncited by 0
  36. 2026Simultaneous Translation between Sign Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Text-Driven Diffusion Model for Sign Language Production
  2. 2023SignDiff: Diffusion Model for American Sign Language Production
  3. 2025Diverse Signer Avatars with Manual and Non-Manual Feature Modelling for Sign Language Production
  4. 2025Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production
  5. 2024MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production
  6. 2026Towards Continuous Sign Language Conversation from Isolated Signs
  7. 2026SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks
  8. 2026SignRefine: Adapting Foundational Video Models for Sign Language Generation