mlsanityData to 14 Oct 2026

Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

arXiv 2410.11190 ↗First posted 15 Oct 2024eess.AS cs.AI cs.CVcited by 109

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

GPT-4o, an all-encompassing model, represents a milestone in the development of large multi-modal language models. It can understand visual, auditory, and textual modalities, directly output audio, and support flexible duplex interaction. Models from the open-source community often achieve some functionalities of GPT-4o, such as visual understanding and voice chat. Nevertheless, training a unified model that incorporates all modalities is challenging due to the complexities of multi-modal data, intricate model architectures, and training processes. In this paper, we introduce Mini-Omni2, a visual-audio assistant capable of providing real-time, end-to-end voice responses to visoin and audio queries. By integrating pretrained visual and auditory encoders, Mini-Omni2 maintains performance in individual modalities. We propose a three-stage training process to align modalities, allowing the language model to handle multi-modal inputs and outputs after training on a limited dataset. For interaction, we introduce a command-based interruption mechanism, enabling more flexible interaction with users. To the best of our knowledge, Mini-Omni2 is one of the closest reproductions of GPT-4o, which have similar form of functionality, and we hope it can offer valuable insights for subsequent research.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 15 Oct 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 59; 2026: 46.

109 papers.

Sort:

By area:

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  2. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  6. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  7. 2025Step-Audio 2 Technical Reportcited by 105
  8. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  9. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  10. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  11. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  12. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 63
38 more
  1. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 60
  2. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  3. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  5. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  6. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
  7. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 34
  8. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 30
  9. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  10. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  11. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  12. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 22
  13. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  14. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 18
  15. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 18
  16. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 17
  17. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  18. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 16
  19. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  20. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  21. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 13
  22. 2025Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systemscited by 13
  23. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  24. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 13
  25. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  26. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  27. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 10
  28. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  29. 2026PASK: Toward Intent-Aware Proactive Agents with Long-Term Memorycited by 9
  30. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluencycited by 8
  31. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chainscited by 8
  32. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  33. 2025MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speechcited by 7
  34. 2026ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decodingcited by 7
  35. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  36. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  37. 2026BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLMcited by 6
  38. 2025AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Taskscited by 5

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
  2. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
  3. 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMs
  4. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
  5. 2026Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
  6. 2026MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model ↗
  7. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
  8. 2025A Preliminary Exploration with GPT-4o Voice Mode