mlsanityData to 7 Oct 2026

SpeechVerse: A Large-scale Generalizable Audio Language Model

arXiv 2405.08295 ↗First posted 14 May 2024cs.CL cs.SD eess.AScited by 62

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have further expanded this capability to perceive multimodal audio and text inputs, but their capabilities are often limited to specific fine-tuned tasks such as automatic speech recognition and translation. We therefore develop SpeechVerse, a robust multi-task training and curriculum learning framework that combines pre-trained speech and text foundation models via a small set of learnable parameters, while keeping the pre-trained models frozen during training. The models are instruction finetuned using continuous latent representations extracted from the speech foundation model to achieve optimal zero-shot performance on a diverse range of speech processing tasks using natural language instructions. We perform extensive benchmarking that includes comparing our model performance against traditional baselines across several datasets and tasks. Furthermore, we evaluate the model's capability for generalized instruction following by testing on out-of-domain datasets, novel prompts, and unseen tasks. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

40 of 54 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
28 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  9. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  10. 2023Listen, Think, and Understandcited by 206
  11. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  12. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  13. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  14. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  15. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  16. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  17. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  18. 2017One Model To Learn Them Allcited by 86
  19. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  20. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  21. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 56
  22. 2023LLaSM: Large Language and Speech Modelcited by 56
  23. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  24. 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understandingcited by 22
  25. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16
  26. 2023Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correctioncited by 14
  27. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  28. 2023Testing Correctness, Fairness, and Robustness of Speech Emotion Recognition Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 31; 2026: 9.

62 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2024Qwen2-Audio Technical Reportcited by 543
  3. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  4. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  5. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  9. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  11. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  12. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
38 more
  1. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  2. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  3. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  5. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 20
  6. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  7. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  8. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognitioncited by 14
  9. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  10. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  11. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  12. 2024PALM: Few-Shot Prompt Learning for Audio Language Modelscited by 8
  13. 2025Attacker's Noise Can Manipulate Your Audio-based LLM in the Real Worldcited by 8
  14. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  15. 2024MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Modelscited by 7
  16. 2024MMFactory: A Universal Solution Search Engine for Vision-Language Taskscited by 7
  17. 2025SPIRIT: Patching Speech Language Models against Jailbreak Attackscited by 7
  18. 2024Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Modelscited by 6
  19. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  20. 2024Zero-resource Speech Translation and Recognition with LLMscited by 6
  21. 2025Soundwave: Less is More for Speech-Text Alignment in LLMscited by 6
  22. 2025Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLMcited by 5
  23. 2024What Are They Doing? Joint Audio-Speech Co-Reasoningcited by 4
  24. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  25. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 4
  26. 2025Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleavingcited by 3
  27. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3
  28. 2024MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoderscited by 2
  29. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challengescited by 2
  30. 2025Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditionscited by 2
  31. 2025A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classificationcited by 2
  32. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2
  33. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Modelcited by 2
  34. 2024Roadmap towards Superhuman Speech Understanding using Large Language Modelscited by 1
  35. 2025SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Modelscited by 1
  36. 2025A Non-autoregressive Model for Joint STT and TTScited by 1
  37. 2025SparQLe: Speech Queries to Text Translation Through LLMscited by 1
  38. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoEcited by 1

50 most cited of 62 All 62 (JSON)