mlsanityData to 14 Oct 2026

Qwen2-Audio Technical Report

arXiv 2407.10759 ↗First posted 15 Jul 2024eess.AS cs.CL cs.LGcited by 560

Not on the map. Why?

Cited by 105 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community.

Key facts. Cited by 560 papers in the map (data through 14 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

30 of 37 references.

References · 30

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
18 more
  1. 2023MusicLM: Generating Music From Textcited by 448
  2. 2019Clotho: An Audio Captioning Datasetcited by 358
  3. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  4. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  5. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  6. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  7. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  8. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  9. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  10. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  11. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  12. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  13. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  14. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  15. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  16. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  17. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16
  18. 2022MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognitioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 46; 2025: 262; 2026: 252.

560 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2025Large Language Diffusion Modelscited by 912
  3. 2025Qwen2.5-Omni Technical Reportcited by 736
  4. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  5. 2025Qwen3-Omni Technical Reportcited by 497
  6. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  7. 2025Kimi-Audio Technical Reportcited by 222
  8. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  9. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  10. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  11. 2026Qwen3.5-Omni Technical Reportcited by 174
  12. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
38 more
  1. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  2. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  3. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  4. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  6. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  7. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  8. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  9. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  10. 2025Step-Audio 2 Technical Reportcited by 105
  11. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  12. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  13. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  14. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  15. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 88
  16. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  17. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  18. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  19. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  20. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 65
  21. 2024Language Model Can Listen While Speakingcited by 63
  22. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  23. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  24. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  25. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  26. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  27. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  28. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  29. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  30. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  31. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 47
  32. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  33. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  34. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  35. 2024Baichuan-Omni Technical Reportcited by 42
  36. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  37. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  38. 2024Foundation Models for Music: A Surveycited by 37

50 most cited of 560 All 560 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
  2. 2024Qwen2.5 Technical Report
  3. 2026Qwen-Audio-3.0-ASR Technical Report
  4. 2024Qwen2 Technical Report
  5. 2025AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
  6. 2025Qwen3 Technical Report
  7. 2026Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm
  8. 2026Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models