mlsanityData to 14 Oct 2026

Qwen2.5-Omni Technical Report

arXiv 2503.20215 ↗First posted 26 Mar 2025cs.CL cs.CV cs.SDcited by 736

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

In this report, we present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. To synchronize the timestamps of video inputs with audio, we organize the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose Thinker-Talker architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming Talker outperforms most existing streaming and non-streaming alternatives in robustness and naturalness.

Key facts. Cited by 736 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 73 of its 85 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

73 of 85 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming29 Aug

Other references · 72

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Flow Matching for Generative Modelingcited by 5,702
  12. 2025Qwen2.5-VL Technical Reportcited by 5,587
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  13. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  14. 2019Towards VQA Models That Can Readcited by 1,718
  15. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  17. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  18. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  19. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  20. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  21. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  22. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  23. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  24. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  25. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  26. 2016A Diagram Is Worth A Dozen Imagescited by 804
  27. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  28. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  29. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  30. 2024Qwen2-Audio Technical Reportcited by 560
  31. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  32. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  33. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  34. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  35. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  36. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  37. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  38. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 289; 2026: 447.

736 papers.

Sort:

By area:

  1. 2025Qwen3-Omni Technical Reportcited by 497
  2. 2025Kimi-Audio Technical Reportcited by 222
  3. 2026Qwen3.5-Omni Technical Reportcited by 174
  4. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  8. 2026Qwen3-TTS Technical Reportcited by 118
  9. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  10. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  11. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  12. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
38 more
  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  3. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  4. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 63
  5. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
  6. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  7. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  8. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 56
  9. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 55
  10. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 54
  11. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  12. 2025Kwai Keye-VL Technical Reportcited by 47
  13. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  14. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  15. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 41
  16. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 41
  17. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  18. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 34
  19. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  20. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  21. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 30
  22. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 30
  23. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 30
  24. 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learningcited by 29
  25. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  26. 2025BusterX++: Towards Unified Cross-Modal AI-Generated Content Detection and Explanation with MLLMcited by 29
  27. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  28. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  29. 2025Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learningcited by 27
  30. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 27
  31. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  32. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 26
  33. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  34. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  35. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 25
  36. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  37. 2025Embodied AI: From LLMs to World Modelscited by 24
  38. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24

50 most cited of 736 All 736 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking
  2. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis
  3. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  4. 2024OMCAT: Omni Context Aware Transformer
  5. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
  6. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generation
  7. 2026OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
  8. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision