mlsanityData to 14 Oct 2026

Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning

arXiv 2409.19510 ↗First posted 29 Sep 2024cs.CLcited by 20

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved significant success in Speech-to-Text Translation (S2TT) tasks. While most existing research has focused on English-centric translation directions, the exploration of many-to-many translation is still limited by the scarcity of parallel data. To address this, we propose a three-stage curriculum learning strategy that leverages the machine translation capabilities of large language models and adapts them to S2TT tasks, enabling effective learning in low-resource settings. We trained MLLMs with varying parameter sizes (3B, 7B, and 32B) and evaluated the proposed strategy using the FLEURS and CoVoST-2 datasets. Experimental results show that the proposed strategy achieves state-of-the-art average performance in 15×14 language pairs, requiring fewer than 10 hours of speech data per language to achieve competitive results. The source code and models are released at https://github.com/yxduir/LLM-SRT.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 34 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

22 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  9. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  11. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  12. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
9 more
  1. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  2. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  3. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  4. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  5. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  6. 2020Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translationcited by 14
  7. 2020Consistent Transcription and Translation of Speechcited by 14
  8. 2019Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Trainingcited by 7
  9. 2023Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 8.

20 papers.

Sort:

By area:

  1. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  2. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  3. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  4. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24
  5. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 14
  6. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Datacited by 7
  7. 2025CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluationcited by 5
  8. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  9. 2025MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languagescited by 3
  10. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?cited by 2
  11. 2026SPD-Faith Bench: Diagnosing and Improving Faithfulness in Chain-of-Thought for Multimodal Large Language Modelscited by 2
  12. 2026Speech LLMs are Contextual Reasoning Transcriberscited by 2
8 more
  1. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  2. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMscited by 0
  3. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0
  4. 2026Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translationcited by 0
  5. 2026Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Modelscited by 0
  6. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoderscited by 0
  7. 2026Nuha-Speech: Building General-Purpose Arabic Speech-LLMscited by 0
  8. 2026Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Speech Translation with Large Language Models: An Industrial Practice
  2. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  3. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  4. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translation
  5. 2023Tuning Large language model for End-to-end Speech Translation
  6. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
  7. 2026CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning
  8. 2026PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs