mlsanityData to 14 Oct 2026

Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs

arXiv 2506.10299 ↗First posted 12 Jun 2025cs.CL cs.SD eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Speech-to-speech translation (S2ST) has been advanced with large language models (LLMs), which are fine-tuned on discrete speech units. In such approaches, modality adaptation from text to speech has been an issue. LLMs are trained on text-only data, which presents challenges to adapt them to speech modality with limited speech-to-speech data. To address the training difficulty, we propose scheduled interleaved speech--text training in this study. We use interleaved speech--text units instead of speech units during training, where aligned text tokens are interleaved at the word level. We gradually decrease the ratio of text as training progresses, to facilitate progressive modality adaptation from text to speech. We conduct experimental evaluations by fine-tuning LLaMA3.2-1B for S2ST on the CVSS dataset. We show that the proposed method consistently improves the translation performances, especially for languages with limited training data.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 40 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

38 of 40 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  9. 2022High Fidelity Neural Audio Compressioncited by 687
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
26 more
  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  3. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  6. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  7. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  8. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  9. 2021Direct speech-to-speech translation with discrete unitscited by 149
  10. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  11. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  12. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  13. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  14. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  15. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 76
  16. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  17. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  18. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  19. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  20. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  21. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  22. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 47
  23. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  24. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  25. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  26. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Cross-Lingual Interleaving for Speech Language Modelscited by 0
  2. 2026Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioningcited by 0
  3. 2026All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translationcited by 0