mlsanityData to 7 Oct 2026

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

arXiv 2407.15415 ↗First posted 22 Jul 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

37 of 52 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  9. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  10. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  11. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  12. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
25 more
  1. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  2. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  3. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  4. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  5. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  6. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  7. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  8. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  9. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  10. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  11. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  12. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  13. 2019Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translationcited by 30
  14. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  15. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  16. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  17. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  18. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  19. 2020Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decodercited by 17
  20. 2019Lattice Transformer for Speech Translationcited by 15
  21. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  22. 2019Self-Attentional Models for Lattice Inputscited by 12
  23. 2020Cascaded Models With Cyclic Feedback For Direct Speech Translationcited by 9
  24. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  25. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 1.

12 papers.

  1. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  2. 2024Language Model Can Listen While Speakingcited by 62
  3. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  4. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 9
  5. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  6. 2024How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Notcited by 4
  7. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?cited by 2
  8. 2025StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservationcited by 1
  9. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignmentcited by 1
  10. 2025LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Datacited by 0
  11. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMscited by 0
  12. 2026Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translationcited by 0