mlsanityData to 7 Oct 2026

Direct Punjabi to English speech translation using discrete units

arXiv 2402.15967 ↗First posted 25 Feb 2024cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Speech-to-speech translation is yet to reach the same level of coverage as text-to-text translation systems. The current speech technology is highly limited in its coverage of over 7000 languages spoken worldwide, leaving more than half of the population deprived of such technology and shared experiences. With voice-assisted technology (such as social robots and speech-to-text apps) and auditory content (such as podcasts and lectures) on the rise, ensuring that the technology is available for all is more important than ever. Speech translation can play a vital role in mitigating technological disparity and creating a more inclusive society. With a motive to contribute towards speech translation research for low-resource languages, our work presents a direct speech-to-speech translation model for one of the Indic languages called Punjabi to English. Additionally, we explore the performance of using a discrete representation of speech called discrete acoustic units as input to the Transformer-based translation model. The model, abbreviated as Unit-to-Unit Translation (U2UT), takes a sequence of discrete units of the source language (the language being translated from) and outputs a sequence of discrete units of the target language (the language being translated to). Our results show that the U2UT model performs better than the Speech-to-Unit Translation (S2UT) model by a 3.69 BLEU score.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

35 of 49 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  12. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
23 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2021A Survey on Neural Speech Synthesiscited by 182
  3. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  4. 2021Direct speech-to-speech translation with discrete unitscited by 148
  5. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  6. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  7. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  8. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  9. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  10. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  11. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  12. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  13. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  14. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  15. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  16. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  17. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  18. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  19. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  20. 2019MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Biblecited by 24
  21. 2023GlobalBench: A Benchmark for Global Progress in Natural Language Processingcited by 8
  22. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5
  23. 2022Fall Detection from Audios with Audio Transformerscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Direct Speech-to-Speech Neural Machine Translation: A Surveycited by 5