mlsanityData to 7 Oct 2026

SignAvatar: Sign Language 3D Motion Reconstruction and Generation

arXiv 2405.07974 ↗First posted 13 May 2024cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Achieving expressive 3D motion reconstruction and automatic generation for isolated sign words can be challenging, due to the lack of real-world 3D sign-word data, the complex nuances of signing motions, and the cross-modal understanding of sign language semantics. To address these challenges, we introduce SignAvatar, a framework capable of both word-level sign language reconstruction and generation. SignAvatar employs a transformer-based conditional variational autoencoder architecture, effectively establishing relationships across different semantic modalities. Additionally, this approach incorporates a curriculum learning strategy to enhance the model's robustness and generalization, resulting in more realistic motions. Furthermore, we contribute the ASL3DWord dataset, composed of 3D joint rotation data for the body, hands, and face, for unique sign words. We demonstrate the effectiveness of SignAvatar through extensive experiments, showcasing its superior reconstruction and automatic generation capabilities. The code and dataset are available on the project page.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

39 of 55 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  9. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  11. 2017End-to-end Recovery of Human Shape and Posecited by 883
  12. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
27 more
  1. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  2. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  5. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  6. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  7. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  8. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  9. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  10. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  11. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  12. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  13. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  14. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  15. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  16. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  17. 2020Progressive Transformers for End-to-End Sign Language Productioncited by 119
  18. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  19. 2021On Self-Contact and Human Posecited by 88
  20. 2020Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimationcited by 81
  21. 2020Adversarial Training for Multi-Channel Sign Language Productioncited by 57
  22. 2020Monocular Real-time Full Body Capture with Inter-part Correlationscited by 53
  23. 2021Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networkscited by 50
  24. 2021Sign Language Production: A Reviewcited by 26
  25. 2021Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognitioncited by 25
  26. 2023Language-guided Human Motion Synthesis with Atomic Actionscited by 17
  27. 2020Revisiting the Continuity of Rotation Representations in Neural Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 8.

14 papers.

Sort:

By area:

  1. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  2. 2024Ig3D: Integrating 3D Face Representations in Facial Expression Inferencecited by 6
  3. 2025Text-Driven 3D Hand Motion Generation from Sign Language Datacited by 5
  4. 2025SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluationcited by 4
  5. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  6. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  7. 2024Studying and Mitigating Biases in Sign Language Understanding Modelscited by 1
  8. 2026Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysiscited by 1
  9. 2026Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Productioncited by 1
  10. 2025HandCraft: Dynamic Sign Generation for Synthetic Data Augmentationcited by 0
  11. 2026InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Readingcited by 0
  12. 2026ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videoscited by 0
2 more
  1. 2026Conditional Collapse in Sign Language Production: A Diagnostic and a Scaling Argumentcited by 0
  2. 2026SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generationcited by 0