mlsanityData to 14 Oct 2026

Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition

arXiv 2312.03668 ↗First posted 6 Dec 2023eess.AS cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

33 of 50 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
21 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  2. 2018Hierarchical Neural Story Generationcited by 1,193
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  4. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2015Attention-Based Models for Speech Recognitioncited by 401
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  9. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  10. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  11. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  12. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  13. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  14. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  15. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  16. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  17. 2017JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesiscited by 58
  18. 2021CTC-based Compression for Direct Speech Translationcited by 45
  19. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 32
  20. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  21. 2023Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 4.

10 papers.

  1. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  2. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 18
  3. 2024Contextualization of ASR with LLM using phonetic retrieval-based augmentationcited by 10
  4. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challengescited by 2
  5. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsamplingcited by 2
  6. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASRcited by 1
  7. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASRcited by 1
  8. 2025Dialect Identification Using Resource-Efficient Fine-Tuning Approachescited by 0
  9. 2026Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Modelscited by 0
  10. 2026Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?cited by 0