# End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMs

arXiv: https://arxiv.org/abs/2510.10329 · HTML: https://mlsanity.com/paper/2510.10329/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [End-to-End Automatic Speech Translation of Audiobooks](https://mlsanity.com/paper/1802.04200/) (2018)
- [Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition](https://mlsanity.com/paper/2306.16007/) (2023)
- [CTC-based Compression for Direct Speech Translation](https://mlsanity.com/paper/2102.01578/) (2021)
- [End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020](https://mlsanity.com/paper/2006.02965/) (2020)
- [Findings of the IWSLT 2024 Evaluation Campaign](https://mlsanity.com/paper/2411.05088/) (2024)

## Built on by (most cited first, 2 of 2)

- [Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning](https://mlsanity.com/paper/2603.06505/) (2026)
- [Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translation](https://mlsanity.com/paper/2609.26536/) (2026)

## Abstract (arXiv)

Speech Translation (ST) is a machine translation task that involves converting speech signals from one language to the corresponding text in another language; this task has two different approaches, namely the traditional cascade and the more recent end-to-end. This paper explores a combined end-to-end architecture of pre-trained speech encoders and Large Language Models (LLMs) for performing both Automatic Speech Recognition (ASR) and ST simultaneously. Experiments with the English-to-German language pair show that our best model not only can achieve better translation results than SeamlessM4T, a large foundational end-to-end, multi-modal translation model, but can also match the performance of a cascaded system with Whisper and NLLB, with up to a score gain of 8% in COMETDA22 metric.