mlsanityData to 14 Oct 2026

Speech Translation with Large Language Models: An Industrial Practice

arXiv 2312.13585 ↗First posted 21 Dec 2023cs.CL cs.SD eess.AScited by 28

Not on the map. Why?

Abstract (arXiv)

Given the great success of large language models (LLMs) across various tasks, in this paper, we introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained LLM. By integrating the large language model (LLM) with a speech encoder and employing multi-task instruction tuning, LLM-ST can produce accurate timestamped transcriptions and translations, even from long audio inputs. Furthermore, our findings indicate that the implementation of Chain-of-Thought (CoT) prompting can yield advantages in the context of LLM-ST. Through rigorous experimentation on English and Chinese datasets, we showcase the exceptional performance of LLM-ST, establishing a new benchmark in the field of speech translation. Demo: https://speechtranslation.github.io/llm-st/.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

30 of 34 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
18 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  4. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  5. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  6. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  7. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  8. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  9. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  10. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  11. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  12. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  13. 2020Rethinking Document-level Neural Machine Translationcited by 32
  14. 2023Do GPTs Produce Less Literal Translations?cited by 30
  15. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  16. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  17. 2023Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentationcited by 10
  18. 2020Towards Making the Most of Context in Neural Machine Translationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 3.

28 papers.

Sort:

By area:

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  2. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  3. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  4. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  5. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  6. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  7. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  8. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  9. 2025Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggerscited by 8
  10. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
  11. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  12. 2025StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Modelcited by 6
16 more
  1. 2024Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languagescited by 5
  2. 2024EMMeTT: Efficient Multimodal Machine Translation Trainingcited by 4
  3. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  4. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  5. 2025CMU's IWSLT 2025 Simultaneous Speech Translation Systemcited by 2
  6. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?cited by 2
  7. 2024CMU's IWSLT 2024 Simultaneous Speech Translation Systemcited by 1
  8. 2025From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognitioncited by 1
  9. 2025Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translationcited by 1
  10. 2025StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservationcited by 1
  11. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learningcited by 0
  12. 2025Toward Machine Interpreting: Lessons from Human Interpreting Studiescited by 0
  13. 2025POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translationcited by 0
  14. 2026EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillationcited by 0
  15. 2026Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translationcited by 0
  16. 2026All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translation
  2. 2023Tuning Large language model for End-to-end Speech Translation
  3. 2024Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning
  4. 2026PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs
  5. 2024What do Large Language Models Need for Machine Translation Evaluation?
  6. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
  7. 2023A Comprehensive Overview of Large Language Models
  8. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation