mlsanityData to 14 Oct 2026

Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

arXiv 2311.07919 ↗First posted 14 Nov 2023eess.AS cs.CL cs.LGcited by 608

Not on the map. Why?

Cited by 87 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.

Key facts. Cited by 608 papers in the map (data through 14 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2022High Fidelity Neural Audio Compressioncited by 687
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  12. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  13. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  14. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  15. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  16. 2019Clotho: An Audio Captioning Datasetcited by 358
  17. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  18. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  19. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  20. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  21. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  22. 2023Listen, Think, and Understandcited by 209
  23. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  24. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  25. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  26. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  27. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  28. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  29. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  30. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  31. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 85
  32. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  33. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  34. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  35. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  36. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  37. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  38. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 159; 2025: 289; 2026: 159.

608 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2025Qwen2.5-Omni Technical Reportcited by 736
  4. 2024Qwen2-Audio Technical Reportcited by 560
  5. 2025Qwen3-Omni Technical Reportcited by 497
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  9. 2025Kimi-Audio Technical Reportcited by 222
  10. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  11. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  12. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
38 more
  1. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  2. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  3. 2026Qwen3.5-Omni Technical Reportcited by 174
  4. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  5. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  6. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  7. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  8. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  9. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  10. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  11. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  12. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  13. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  14. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  15. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  16. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  17. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  18. 2025Step-Audio 2 Technical Reportcited by 105
  19. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  20. 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 103
  21. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  22. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  23. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  24. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  25. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  26. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  27. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  28. 2024Best-of-N Jailbreakingcited by 76
  29. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  30. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  31. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  32. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  33. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  34. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  35. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  36. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 65
  37. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  38. 2024Language Model Can Listen While Speakingcited by 63

50 most cited of 608 All 608 (JSON)