mlsanityData to 7 Oct 2026

A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars

arXiv 2401.04730 ↗First posted 9 Jan 2024cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

The objective of this paper is to develop a functional system for translating spoken languages into sign languages, referred to as Spoken2Sign translation. The Spoken2Sign task is orthogonal and complementary to traditional sign language to spoken language (Sign2Spoken) translation. To enable Spoken2Sign translation, we present a simple baseline consisting of three steps: 1) creating a gloss-video dictionary using existing Sign2Spoken benchmarks; 2) estimating a 3D sign for each sign video in the dictionary; 3) training a Spoken2Sign model, which is composed of a Text2Gloss translator, a sign connector, and a rendering module, with the aid of the yielded gloss-3D sign dictionary. The translation results are then displayed through a sign avatar. As far as we know, we are the first to present the Spoken2Sign task in an output format of 3D signs. In addition to its capability of Spoken2Sign translation, we also demonstrate that two by-products of our approach-3D keypoint augmentation and multi-view understanding-can assist in keypoint-based sign language understanding. Code and models are available at https://github.com/FangyunWei/SLRT.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 61 of its 95 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

61 of 95 references.

References · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2017End-to-end Recovery of Human Shape and Posecited by 883
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  11. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  12. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
38 more
  1. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  2. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  3. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  4. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  5. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  6. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  7. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
  8. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  9. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 122
  10. 2020Progressive Transformers for End-to-End Sign Language Productioncited by 119
  11. 2018MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Languagecited by 118
  12. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  13. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 104
  14. 2021Skeleton Aware Multi-modal Sign Language Recognitioncited by 101
  15. 2019DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Comparecited by 100
  16. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  17. 2021Visual Alignment Constraint for Continuous Sign Language Recognitioncited by 79
  18. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
  19. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  20. 2018How good is my GAN?cited by 74
  21. 2020Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognitioncited by 74
  22. 2020Multi-channel Transformers for Multi-articulatory Sign Language Translationcited by 71
  23. 2022Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Productioncited by 61
  24. 2020Transferring Cross-domain Knowledge for Video Sign Language Recognitioncited by 60
  25. 2023SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understandingcited by 59
  26. 2020Fully Convolutional Networks for Continuous Sign Language Recognitioncited by 58
  27. 2020Adversarial Training for Multi-Channel Sign Language Productioncited by 57
  28. 2023Continuous Sign Language Recognition with Correlation Networkcited by 56
  29. 2021Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networkscited by 50
  30. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  31. 2023Natural Language-Assisted Sign Language Recognitioncited by 46
  32. 2023CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignmentcited by 44
  33. 2023Sign Language Translation from Instructional Videoscited by 39
  34. 2020Watch, read and lookup: learning to spot signs from multiple supervisorscited by 35
  35. 2023BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenizationcited by 35
  36. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  37. 2022Self-Emphasizing Network for Continuous Sign Language Recognitioncited by 33
  38. 2023CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learningcited by 32

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 9.

18 papers.

Sort:

By area:

  1. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  2. 2024Towards Online Continuous Sign Language Recognition and Translationcited by 13
  3. 2024Pose-Guided Fine-Grained Sign Language Video Generationcited by 8
  4. 2025Meaningful Pose-Based Sign Language Evaluationcited by 6
  5. 2025SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Workcited by 4
  6. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 4
  7. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  8. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  9. 2025SIGNER: Temporally Grounded Sign Language Generation via Time-Resolved Conditioningcited by 1
  10. 2025Motion is the Choreographer: Learning Latent Pose Dynamics for Seamless Sign Language Generationcited by 1
  11. 2025Introducing A Bangla Sentence - Gloss Pair Dataset for Bangla Sign Language Translation and Researchcited by 1
  12. 2026Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Productioncited by 1
6 more
  1. 2026Towards Continuous Sign Language Conversation from Isolated Signscited by 0
  2. 2026Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standardscited by 0
  3. 2026From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigationcited by 0
  4. 2026Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Productioncited by 0
  5. 2026PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmarkcited by 0
  6. 2026SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scalecited by 0