mlsanityData to 7 Oct 2026

Qwen3-Omni Technical Report

arXiv 2509.17765 ↗First posted 22 Sep 2025cs.CL cs.AI cs.CVcited by 451

Not on the map. Why?

Cited by 94 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. Qwen3-Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Qwen3-Omni adopts a Thinker-Talker MoE architecture that unifies perception and generation across text, images, audio, and video, yielding fluent text and natural real-time speech. It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages. To reduce first-packet latency in streaming synthesis, Talker autoregressively predicts discrete speech codecs using a multi-codebook scheme. Leveraging the representational capacity of these codebooks, we replace computationally intensive block-wise diffusion with a lightweight causal ConvNet, enabling streaming from the first codec frame. In cold-start settings, Qwen3-Omni achieves a theoretical end-to-end first-packet latency of 234 ms. To further strengthen multimodal reasoning, we introduce a Thinking model that explicitly reasons over inputs from any modality. Since the research community currently lacks a general-purpose audio captioning model, we fine-tuned Qwen3-Omni-30B-A3B to obtain Qwen3-Omni-30B-A3B-Captioner, which produces detailed, low-hallucination captions for arbitrary audio inputs. Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner are publicly released under the Apache 2.0 license.

Key facts. Cited by 451 papers in the map (data through 7 Oct 2026). Builds on 55 of its 66 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

55 of 66 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2016A Diagram Is Worth A Dozen Imagescited by 793
  11. 2025Group Sequence Policy Optimizationcited by 788
  12. 2025Qwen2.5-Omni Technical Reportcited by 704
  13. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  14. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  15. 2024Qwen2-Audio Technical Reportcited by 543
  16. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  17. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  18. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  19. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  20. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  21. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  22. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  23. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 210
  24. 2023Teaching CLIP to Count to Tencited by 194
  25. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  26. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  27. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  28. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  29. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  30. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  31. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  32. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  33. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  34. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  35. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 95
  36. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  37. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  38. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 54; 2026: 397.

451 papers.

Sort:

By area:

  1. 2026Qwen3.5-Omni Technical Reportcited by 146
  2. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  3. 2026Qwen3-ASR Technical Reportcited by 105
  4. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  6. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 46
  7. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  8. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  9. 2025Step-Audio-R1 Technical Reportcited by 35
  10. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  11. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  12. 2025LongCat-Flash-Omni Technical Reportcited by 23
38 more
  1. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
  2. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  3. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 21
  4. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  5. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  6. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  7. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  8. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 16
  9. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  10. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  11. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  12. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  13. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 13
  14. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 12
  15. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  16. 2026Proact-VL: A Proactive VideoLLM for Real-Time AI Companionscited by 12
  17. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  18. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 11
  19. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 11
  20. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
  21. 2026GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Taskscited by 11
  22. 2026ThinkRL-Edit: Thinking in Reinforcement Learning for Reasoning-Centric Image Editingcited by 10
  23. 2026Where Do the Joules Go? Diagnosing Inference Energy Consumptioncited by 10
  24. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 10
  25. 2026Closing the Modality Reasoning Gap for Speech Large Language Modelscited by 9
  26. 2026ERNIE 5.0 Technical Reportcited by 9
  27. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  28. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 9
  29. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 9
  30. 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guardcited by 8
  31. 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
  32. 2025LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillationcited by 8
  33. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 8
  34. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8
  35. 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 8
  36. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  37. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Modelscited by 7
  38. 2026Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Controlcited by 7

50 most cited of 451 All 451 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generation
  2. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
  3. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  4. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis
  5. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
  6. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
  7. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  8. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching