mlsanityData to 7 Oct 2026

Comparing Discrete and Continuous Space LLMs for Speech Recognition

arXiv 2409.00800 ↗First posted 1 Sep 2024cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

22 of 32 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
10 more
  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  4. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  5. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  6. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  7. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  8. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  9. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  10. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  2. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  3. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  4. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  5. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 7
  6. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  7. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modelingcited by 6
  8. 2026Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Modelscited by 5
  9. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  10. 2025LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenizationcited by 1
  11. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASRcited by 1
  12. 2026ReHear: Iterative Pseudo-Label Refinement for Semi-Supervised Speech Recognition via Audio Large Language Modelscited by 1
3 more
  1. 2026Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translationcited by 0
  2. 2026HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Modelscited by 0
  3. 2026Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMscited by 0