mlsanityData to 14 Oct 2026

Phonology-Guided Speech-to-Speech Translation for African Languages

arXiv 2410.23323 ↗First posted 30 Oct 2024eess.AS cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a prosody-guided framework for speech-to-speech translation (S2ST) that aligns and translates speech without transcripts by leveraging cross-linguistic pause synchrony. Analyzing a 6,000-hour East African news corpus spanning five languages, we show that within-phylum language pairs exhibit 30--40% lower pause variance and over 3× higher onset/offset correlation compared to cross-phylum pairs. These findings motivate SPaDA, a dynamic-programming alignment algorithm that integrates silence consistency, rate synchrony, and semantic similarity. SPaDA improves alignment F1 by +3--4 points and eliminates up to 38% of spurious matches relative to greedy VAD baselines. Using SPaDA-aligned segments, we train SegUniDiff, a diffusion-based S2ST model guided by external gradients from frozen semantic and speaker encoders. SegUniDiff matches an enhanced cascade in BLEU (30.3 on CVSS-C vs. 28.9 for UnitY), reduces speaker error rate (EER) from 12.5% to 5.3%, and runs at an RTF of 1.02. To support evaluation in low-resource settings, we also release a three-tier, transcript-free BLEU suite (M1--M3) that correlates strongly with human judgments. Together, our results show that prosodic cues in multilingual speech provide a reliable scaffold for scalable, non-autoregressive S2ST.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

25 of 50 references.

References · 25

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  10. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  12. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
13 more
  1. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  2. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  3. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  4. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  5. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  6. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  7. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  8. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  9. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  10. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  11. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  12. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17
  13. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5

Built on by

Not cited yet.