Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation
Not on the map. Why?
Abstract (arXiv)
The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using n-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 21 May 2025.
Builds on
20 of 28 references.
References · 20
- 2017Attention Is All You Needcited by 60,361
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
- 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
- 2021Direct speech-to-speech translation with discrete unitscited by 149
- 2021Textless Speech-to-Speech Translation on Real Datacited by 121
- 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
- 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
- 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
8 more
- 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
- 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
- 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
- 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
- 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
- 2023Acoustic BPE for Speech Generation with Discrete Tokenscited by 15
- 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
- 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.