mlsanityData to 14 Oct 2026

On decoder-only architecture for speech-to-text and large language model integration

arXiv 2307.03917 ↗First posted 8 Jul 2023eess.AS cs.CL cs.SDcited by 145

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech signals into LLMs has not been explored well. The "decoder-only" architecture has also not been well studied for speech processing tasks. In this research, we introduce Speech-LLaMA, a novel approach that effectively incorporates acoustic information into text-based large language models. Our method leverages Connectionist Temporal Classification and a simple audio encoder to map the compressed acoustic features to the continuous semantic space of the LLM. In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines, highlighting the potential advantages of decoder-only models for speech-to-text conversion.

Key facts. Cited by 145 papers in the map (data through 14 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 8 Jul 2023.

Builds on

35 of 43 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
23 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2020GLU Variants Improve Transformercited by 1,804
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  8. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  9. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  10. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  13. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  14. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  15. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  16. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  17. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  18. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  19. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  20. 2021CTC-based Compression for Direct Speech Translationcited by 45
  21. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  22. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  23. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 70; 2025: 40; 2026: 23.

145 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  4. 2024Qwen2-Audio Technical Reportcited by 560
  5. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  6. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  7. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  9. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  10. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  11. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  12. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 83
38 more
  1. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  2. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  3. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  4. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  5. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  6. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  7. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  8. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  9. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  10. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  11. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  12. 2023Can Whisper perform speech-based in-context learning?cited by 35
  13. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  14. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  15. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 31
  16. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  17. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 31
  18. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  19. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  20. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 29
  21. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  22. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  23. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  24. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  25. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 21
  26. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 19
  27. 2024CTC-Assisted LLM-Based Contextual ASRcited by 19
  28. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  29. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  30. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  31. 2024Generative Expressive Conversational Speech Synthesiscited by 16
  32. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  33. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 15
  34. 2024Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasetscited by 13
  35. 2024Decoder-only Architecture for Streaming End-to-end Speech Recognitioncited by 13
  36. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  37. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  38. 2024Efficient Streaming LLM for Speech Recognitioncited by 12

50 most cited of 145 All 145 (JSON)