mlsanityData to 14 Oct 2026

Ming-Omni: A Unified Multimodal Model for Perception and Generation

arXiv 2506.09344 ↗First posted 11 Jun 2025cs.AI cs.CL cs.CVcited by 66

Not on the map. Why?

Abstract (arXiv)

We propose Ming-Omni, a unified multimodal model capable of processing images, text, audio, and video, while demonstrating strong proficiency in both speech and image generation. Ming-Omni employs dedicated encoders to extract tokens from different modalities, which are then processed by Ling, an MoE architecture equipped with newly proposed modality-specific routers. This design enables a single model to efficiently process and fuse multimodal inputs within a unified framework, thereby facilitating diverse tasks without requiring separate models, task-specific fine-tuning, or structural redesign. Importantly, Ming-Omni extends beyond conventional multimodal models by supporting audio and image generation. This is achieved through the integration of an advanced audio decoder for natural-sounding speech and Ming-Lite-Uni for high-quality image generation, which also allow the model to engage in context-aware chatting, perform text-to-speech conversion, and conduct versatile image editing. Our experimental results showcase Ming-Omni offers a powerful solution for unified perception and generation across all modalities. Notably, our proposed Ming-Omni is the first open-source model we are aware of to match GPT-4o in modality support, and we release all code and model weights to encourage further research and development in the community.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 77 of its 96 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

77 of 96 references.

References · 77

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
38 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  2. 2025Qwen2.5-Omni Technical Reportcited by 736
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  5. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  6. 2024Qwen2-Audio Technical Reportcited by 560
  7. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  11. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  12. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  13. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  14. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  15. 2025Kimi-VL Technical Reportcited by 388
  16. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  17. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  18. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  19. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
  20. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  21. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  22. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  23. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  24. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  25. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 236
  26. 2025Transfer between Modalities with MetaQueriescited by 236
  27. 2025Kimi-Audio Technical Reportcited by 222
  28. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  29. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  30. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  31. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  32. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  33. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  34. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  35. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  36. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 115
  37. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  38. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 42.

66 papers.

Sort:

By area:

  1. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  2. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  3. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24
  4. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
  5. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 22
  6. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 22
  7. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  8. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  9. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 17
  10. 2025M2-Reasoning: Empowering MLLMs with Unified General and Spatial Reasoningcited by 16
  11. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  12. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 12
38 more
  1. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligencecited by 11
  2. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 11
  3. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 11
  4. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  5. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 8
  6. 2026Context Unrolling in Omni Modelscited by 8
  7. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chainscited by 8
  8. 2026Toward Native Multimodal Modeling: A Roadmapcited by 7
  9. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  10. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  11. 2025FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoningcited by 5
  12. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  13. 20253SGen: Unified Subject, Style, and Structure-Driven Image Generation with Adaptive Task-specific Memorycited by 5
  14. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  15. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 5
  16. 2025UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoEcited by 4
  17. 2025MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generationcited by 4
  18. 2026HyperCLOVA X 8B Omnicited by 4
  19. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Modelscited by 4
  20. 2026OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistantscited by 4
  21. 2026MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generationcited by 4
  22. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  23. 2025Target-aware Image Editing via Cycle-consistent Constraintscited by 3
  24. 2026Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detectioncited by 3
  25. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 3
  26. 2026Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoningcited by 3
  27. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generationcited by 2
  28. 2026Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Modelscited by 2
  29. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 2
  30. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  31. 2026When Vision Speaks for Soundcited by 2
  32. 2026AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioningcited by 2
  33. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  34. 2025MolSculpt: Sculpting 3D Molecular Geometries from Chemical Syntaxcited by 1
  35. 2026Eureka-Audio: Triggering Audio Intelligence in Compact Language Modelscited by 1
  36. 2026CoEditor++: Instruction-based Visual Editing via Cognitive Reasoningcited by 1
  37. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 1
  38. 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialectscited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction
  2. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception
  3. 2024Baichuan-Omni Technical Report
  4. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
  5. 2025Qwen3-Omni Technical Report
  6. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  7. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Model
  8. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM