mlsanityData to 14 Oct 2026

LLaSM: Large Language and Speech Model

arXiv 2308.15930 ↗First posted 30 Aug 2023cs.CL cs.LG cs.SDcited by 58

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal large language models have garnered significant interest recently. Though, most of the works focus on vision-language multi-modal models providing strong capabilities in following vision-and-language instructions. However, we claim that speech is also an important modality through which humans interact with the world. Hence, it is crucial for a general-purpose assistant to be able to follow multi-modal speech-and-language instructions. In this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an end-to-end trained large multi-modal speech-language model with cross-modal conversational abilities, capable of following speech-and-language instructions. Our early experiments show that LLaSM demonstrates a more convenient and natural way for humans to interact with artificial intelligence. Specifically, we also release a large Speech Instruction Following dataset LLaSM-Audio-Instructions. Code and demo are available at https://github.com/LinkSoul-AI/LLaSM and https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions dataset is available at https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 26 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

18 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
5 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  2. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  4. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  5. 2023Emu: Generative Pretraining in Multimodalitycited by 157

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 29; 2025: 16; 2026: 7.

58 papers.

Sort:

By area:

  1. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  2. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 216
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  4. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  5. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  6. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  7. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  8. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  9. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  10. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  11. 2024Best-of-N Jailbreakingcited by 76
  12. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
38 more
  1. 2024A Full-duplex Speech Dialogue Scheme Based On Large Language Modelscited by 40
  2. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  3. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  4. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  5. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  6. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  7. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 31
  8. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  9. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  10. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  11. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  12. 2025MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Meshcited by 20
  13. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 18
  14. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 18
  15. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 11
  16. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  17. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 9
  18. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  19. 2024SALSA: Speedy ASR-LLM Synchronous Aggregationcited by 8
  20. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 8
  21. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Datacited by 7
  22. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 7
  23. 2025Megrez-Omni Technical Reportcited by 7
  24. 2024SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioningcited by 6
  25. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  26. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  27. 2024A Unified Framework for Motion Reasoning and Generation in Human Interactioncited by 5
  28. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  29. 2024The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggerscited by 3
  30. 2025Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleavingcited by 3
  31. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3
  32. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injectioncited by 3
  33. 2024MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoderscited by 2
  34. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  35. 2023SkyMath: Technical Reportcited by 1
  36. 2024Roadmap towards Superhuman Speech Understanding using Large Language Modelscited by 1
  37. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignmentcited by 1
  38. 2026Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMscited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Review of Multi-Modal Large Language and Vision Models
  2. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  3. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  4. 2023A Comprehensive Overview of Large Language Models
  5. 2026Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
  6. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  7. 2024SoupLM: Model Integration in Large Language and Multi-Modal Models
  8. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models