mlsanityData to 7 Oct 2026

COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning

arXiv 2311.02248 ↗First posted 3 Nov 2023cs.CL cs.AI eess.AScited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present a cost-effective method to integrate speech into a large language model (LLM), resulting in a Contextual Speech Model with Instruction-following/in-context-learning Capabilities (COSMIC) multi-modal LLM. Using GPT-3.5, we generate Speech Comprehension Test Question-Answer (SQA) pairs from speech transcriptions for supervised instruction tuning. With under 30 million trainable parameters and only 450 hours of English speech data, COSMIC demonstrates emerging capabilities in instruction-following and in-context learning. Equipped with such capabilities, COSMIC achieves a maximum 33.18 BLEU score in 0-shot EN-to-X speech to text translation (S2TT) and a significant boost in the 1-shot setting. Additionally, there is an average 25.8% relative Word Error Rate (WER) reduction for 1-shot cross-domain adaptation. COSMIC exhibits a significant automatic speech recognition (ASR) accuracy gain in contextual biasing tasks due to its instruction-following capability.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 29 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

26 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
13 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  4. 2023Listen, Think, and Understandcited by 206
  5. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  6. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  7. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  8. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  9. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  10. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  11. 2023Can Whisper perform speech-based in-context learning?cited by 35
  12. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  13. 2023Instruction-Following Speech Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 6; 2026: 6.

30 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  3. 2024Language Model Can Listen While Speakingcited by 62
  4. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  5. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  6. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  7. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  8. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  9. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  10. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  11. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  12. 2024SpeechQE: Estimating the Quality of Direct Speech Translationcited by 10
18 more
  1. 2024Instruction Data Generation and Unsupervised Adaptation for Speech Language Modelscited by 9
  2. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  3. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  4. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 8
  5. 2024SALSA: Speedy ASR-LLM Synchronous Aggregationcited by 8
  6. 2024Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalitiescited by 7
  7. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  8. 2024Self-Powered LLM Modality Expansion for Large Speech-Text Modelscited by 4
  9. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  10. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  11. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  12. 2024Roadmap towards Superhuman Speech Understanding using Large Language Modelscited by 1
  13. 2026MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learningcited by 1
  14. 2026Measuring the Redundancy of Decoder Layers in SpeechLLMscited by 0
  15. 2026A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfercited by 0
  16. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0
  17. 2026In-Context Adaptation of Encoder-Decoder Models in Speech Recognitioncited by 0
  18. 2026When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Testcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
  2. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning
  3. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  4. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
  5. 2025COSMOS: Predictable and Cost-Effective Adaptation of LLMs
  6. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition
  7. 2023FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning
  8. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs