mlsanityData to 14 Oct 2026

MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech

arXiv 2509.25131 ↗First posted 29 Sep 2025cs.SD cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

We present MGM-Omni, a unified Omni LLM for omni-modal understanding and expressive, long-horizon speech generation. Unlike cascaded pipelines that isolate speech synthesis, MGM-Omni adopts a "brain-mouth" design with a dual-track, token-based architecture that cleanly decouples multimodal reasoning from real-time speech generation. This design enables efficient cross-modal interaction and low-latency, streaming speech generation. For understanding, a unified training strategy coupled with a dual audio encoder design enables long-form audio perception across diverse acoustic conditions. For generation, a chunk-based parallel decoding scheme narrows the text speech token-rate gap, accelerating inference and supporting streaming zero-shot voice cloning with stable timbre over extended durations. Compared to concurrent work, MGM-Omni achieves these capabilities with markedly data-efficient training. Extensive experiments demonstrate that MGM-Omni outperforms existing open source models in preserving timbre identity across extended sequences, producing natural and context-aware speech, and achieving superior long-form audio and omnimodal understanding. MGM-Omni establishes an efficient, end-to-end paradigm for omnimodal understanding and controllable, personalised long-horizon speech generation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

45 of 55 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
33 more
  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2025Qwen2.5-Omni Technical Reportcited by 736
  9. 2024Qwen2-Audio Technical Reportcited by 560
  10. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  13. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  14. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  15. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  16. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  17. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  18. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  19. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  20. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  21. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  22. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  23. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  24. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  25. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  26. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  27. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 109
  28. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  29. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  30. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  31. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  32. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  33. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
  2. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
  3. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  4. 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMs
  5. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
  6. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
  7. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
  8. 2026Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models