mlsanityData to 7 Oct 2026

Reconstructing Signing Avatars From Video Using Linguistic Priors

arXiv 2304.10482 ↗First posted 20 Apr 2023cs.CV cs.GRcited by 12

Not on the map. Why?

Abstract (arXiv)

Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. Video dictionaries of isolated signs are a core SL learning tool. Replacing these with 3D avatars can aid learning and enable AR/VR applications, improving access to technology and online media. However, little work has attempted to estimate expressive 3D avatars from SL video; occlusion, noise, and motion blur make this task difficult. We address this by introducing novel linguistic priors that are universally applicable to SL and provide constraints on 3D hand pose that help resolve ambiguities within isolated signs. Our method, SGNify, captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos. We evaluate SGNify quantitatively by using a commercial motion-capture system to compute 3D avatars synchronized with monocular video. SGNify outperforms state-of-the-art 3D body-pose- and shape-estimation methods on SL videos. A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos. Code and data are available at hrefhttp://sgnify.is.tue.mpg.desgnify.is.tue.mpg.de.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 67 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

35 of 67 references.

References · 35

  1. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  2. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  3. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  4. 2017End-to-end Recovery of Human Shape and Posecited by 883
  5. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  6. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  7. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  8. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  9. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  10. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  11. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  12. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
23 more
  1. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  2. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  3. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  4. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  5. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  6. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  7. 2020Deep Learning-Based Human Pose Estimation: A Surveycited by 205
  8. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  9. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  10. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  11. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  12. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  13. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  14. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  15. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  16. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  17. 20233D Human Pose Estimation via Intuitive Physicscited by 100
  18. 2021On Self-Contact and Human Posecited by 88
  19. 2020Neural Descent for Visual 3D Human Pose and Shapecited by 60
  20. 2022Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videoscited by 50
  21. 2021Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognitioncited by 25
  22. 2021Sign Segmentation with Changepoint-Modulated Pseudo-Labellingcited by 10
  23. 2020Independent Sign Language Recognition with 3D Body, Hands, and Face Reconstructioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 3.

12 papers.

  1. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  2. 2023Neural Sign Actors: A diffusion model for 3D sign language production from textcited by 48
  3. 2023SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmarkcited by 29
  4. 2024Expressive Gaussian Human Avatars from Monocular RGB Videocited by 21
  5. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  6. 2024A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatarscited by 18
  7. 2025Text-Driven 3D Hand Motion Generation from Sign Language Datacited by 5
  8. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  9. 2025DexAvatar: 3D Sign Language Reconstruction with Hand and Body Pose Priorscited by 1
  10. 2025Contact-Aware Refinement of Human Pose Pseudo-Ground Truth via Bioimpedance Sensingcited by 0
  11. 2026Tamaththul3D: High-Fidelity 3D Saudi Sign Language Avatars from Monocular Videocited by 0
  12. 2026PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmarkcited by 0