MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech
Not on the map. Why?
Abstract (arXiv)
We present MGM-Omni, a unified Omni LLM for omni-modal understanding and expressive, long-horizon speech generation. Unlike cascaded pipelines that isolate speech synthesis, MGM-Omni adopts a "brain-mouth" design with a dual-track, token-based architecture that cleanly decouples multimodal reasoning from real-time speech generation. This design enables efficient cross-modal interaction and low-latency, streaming speech generation. For understanding, a unified training strategy coupled with a dual audio encoder design enables long-form audio perception across diverse acoustic conditions. For generation, a chunk-based parallel decoding scheme narrows the text speech token-rate gap, accelerating inference and supporting streaming zero-shot voice cloning with stable timbre over extended durations. Compared to concurrent work, MGM-Omni achieves these capabilities with markedly data-efficient training. Extensive experiments demonstrate that MGM-Omni outperforms existing open source models in preserving timbre identity across extended sequences, producing natural and context-aware speech, and achieving superior long-form audio and omnimodal understanding. MGM-Omni establishes an efficient, end-to-end paradigm for omnimodal understanding and controllable, personalised long-horizon speech generation.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 29 Sep 2025.
Builds on
45 of 55 references.
References · 45
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Visual Instruction Tuningcited by 7,869
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
33 more
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
- 2025s1: Simple test-time scalingcited by 1,341
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2024Qwen2-Audio Technical Reportcited by 560
- 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
- 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
- 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
- 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
- 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
- 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
- 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
- 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
- 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
- 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
- 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
- 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
- 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
- 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 109
- 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
- 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
- 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
- 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
- 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
- 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
Built on by
Citing papers per year: 2025: 1; 2026: 6.
7 papers.
- 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
- 2026Fish Audio S2 Technical Reportcited by 9
- 2026UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Modelscited by 6
- 2026OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Modelscited by 1
- 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
- 2025ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Modelscited by 0
- 2026Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Taskcited by 0
- 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges18 Aug 2026
- 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction30 Apr 2026
- 2026OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models25 Mar 2026
- 2026Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Task16 Mar 2026
- 2026Fish Audio S2 Technical Report9 Mar 2026
- 2026UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models4 Jan 2026
- 2025ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models12 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
- 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
- 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
- 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMs
- 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
- 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
- 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
- 2026Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models