mlsanityData to 14 Oct 2026

IsoSignVid2Aud: Sign Language Video to Audio Conversion without Text Intermediaries

arXiv 2510.07837 ↗First posted 9 Oct 2025cs.CV cs.MM cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Sign language to spoken language audio translation is important to connect the hearing- and speech-challenged humans with others. We consider sign language videos with isolated sign sequences rather than continuous grammatical signing. Such videos are useful in educational applications and sign prompt interfaces. Towards this, we propose IsoSignVid2Aud, a novel end-to-end framework that translates sign language videos with a sequence of possibly non-grammatic continuous signs to speech without requiring intermediate text representation, providing immediate communication benefits while avoiding the latency and cascading errors inherent in multi-stage translation systems. Our approach combines an I3D-based feature extraction module with a specialized feature transformation network and an audio generation pipeline, utilizing a novel Non-Maximal Suppression (NMS) algorithm for the temporal detection of signs in non-grammatic continuous sequences. Experimental results demonstrate competitive performance on ASL-Citizen-1500 and WLASL-100 datasets with Top-1 accuracies of 72.01% and 78.67%, respectively, and audio quality metrics (PESQ: 2.67, STOI: 0.73) indicating intelligible speech output. Code is available at: https://github.com/BheeshmSharma/IsoSignVid2AudAIMLsystems-2025.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 27 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

15 of 27 references.

References · 15

  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2015Fast R-CNNcited by 2,933
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 256
  9. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 218
  10. 2021Visual Alignment Constraint for Continuous Sign Language Recognitioncited by 80
  11. 2020Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognitioncited by 74
  12. 2018Neural Sign Language Translation based on Human Keypoint Estimationcited by 65
3 more
  1. 2023ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognitioncited by 34
  2. 2022StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognitioncited by 9
  3. 2021Word-level Sign Language Recognition with Multi-stream Neural Networks Focusing on Local Regions and Skeletal Informationcited by 3

Built on by

Not cited yet.