mlsanityData to 7 Oct 2026

AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs

arXiv 2311.06753 ↗First posted 12 Nov 2023cs.CL cs.AIcited by 59

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

In this work, we extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities while maintaining the wide range of original LLM capabilities, without using any carefully curated paired data. The resulting end-to-end model, named AudioChatLlama, can utilize audio prompts as a replacement for text and sustain a conversation. Such a model also has extended cross-modal capabilities such as being able to perform spoken question answering (QA), speech translation, and audio summarization amongst many other closed and open-domain tasks. This is unlike prior approaches in speech, in which LLMs are extended to handle audio for a limited number of pre-designated tasks. On both synthesized and recorded speech QA test sets, evaluations show that our end-to-end approach is on par with or outperforms cascaded systems (speech recognizer + LLM) in terms of modeling the response to a prompt. Furthermore, unlike cascades, our approach can interchange text and audio modalities and intrinsically utilize prior context in a conversation to provide better results.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 37 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

30 of 37 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
18 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2021Ethical and social risks of harm from Language Modelscited by 957
  7. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  9. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2021Alignment of Language Agentscited by 167
  12. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  13. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  14. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  15. 2019Speech Recognition with Augmented Synthesized Speechcited by 60
  16. 2019Generating Synthetic Audio Data for Attention-Based Speech Recognition Systemscited by 46
  17. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  18. 2023Towards Selection of Text-to-speech Data to Augment ASR Trainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 24; 2026: 14.

59 papers.

Sort:

By area:

  1. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  4. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  5. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  6. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  7. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  8. 2024Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuancescited by 27
  9. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  10. 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 20
  11. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  12. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 16
38 more
  1. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  2. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  3. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  4. 2024Efficient Streaming LLM for Speech Recognitioncited by 10
  5. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  6. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  7. 2023BrainLLM: Generative Language Decoding from Brain Recordingscited by 7
  8. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  9. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 7
  10. 2025WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearablescited by 7
  11. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
  12. 2024Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairnesscited by 6
  13. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6
  14. 2025Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech Datacited by 6
  15. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  16. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  17. 2024Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languagescited by 4
  18. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  19. 2025Trick or Neat: Adversarial Ambiguity and Language Model Evaluationcited by 3
  20. 2025Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognitioncited by 3
  21. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3
  22. 2024Towards Diverse and Efficient Audio Captioning via Diffusion Modelscited by 2
  23. 2024CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASRcited by 2
  24. 2024Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modalitycited by 2
  25. 2025Speech Translation Refinement using Large Language Modelscited by 2
  26. 2025UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speechcited by 2
  27. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  28. 2025Optimal Transport Regularization for Speech Text Alignment in Spoken Language Modelscited by 2
  29. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuningcited by 2
  30. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  31. 2024Roadmap towards Superhuman Speech Understanding using Large Language Modelscited by 1
  32. 2025Spoken question answering for visual queriescited by 1
  33. 2025AC/DC: LLM-based Audio Comprehension via Dialogue Continuationcited by 1
  34. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignmentcited by 1
  35. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  36. 2025SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answeringcited by 0
  37. 2025GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLMcited by 0
  38. 2025Dual Information Speech Language Models for Emotional Conversationscited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation
  2. 2025Can Speech LLMs Think while Listening?
  3. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM
  4. 2025AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
  5. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue
  6. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages
  7. 2023LLaSM: Large Language and Speech Model
  8. 2024Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM