# MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction

arXiv: https://arxiv.org/abs/2604.27393 · HTML: https://mlsanity.com/paper/2604.27393/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 61 of its 75 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 61)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)

## Built on by (most cited first, 30 of 78)

- [OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding](https://mlsanity.com/paper/2605.18577/) (2026)
- [Toward Native Multimodal Modeling: A Roadmap](https://mlsanity.com/paper/2605.25343/) (2026)
- [Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models](https://mlsanity.com/paper/2606.25041/) (2026)
- [Orca: The World is in Your Mind](https://mlsanity.com/paper/2606.30534/) (2026)
- [Realtime-Venus: A full-duplex interaction system with asynchronous delegation](https://mlsanity.com/paper/2609.13814/) (2026)
- [NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities](https://mlsanity.com/paper/2609.21967/) (2026)
- [Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning](https://mlsanity.com/paper/2605.28192/) (2026)
- [AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression](https://mlsanity.com/paper/2606.24286/) (2026)
- [OmniReasoner: Thinking with Long Audio-Video via Native Tool Use](https://mlsanity.com/paper/2607.19339/) (2026)
- [Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning](https://mlsanity.com/paper/2607.20166/) (2026)
- [OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2608.03812/) (2026)
- [StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding](https://mlsanity.com/paper/2608.05703/) (2026)
- [Multimodal Duplex Interaction Agent](https://mlsanity.com/paper/2609.08977/) (2026)
- [When Vision Speaks for Sound](https://mlsanity.com/paper/2605.16403/) (2026)
- [ReUnit: Multi-Granularity Visual Unitization for Long Video Understanding](https://mlsanity.com/paper/2606.12125/) (2026)
- [Wan-Streamer v0.2: Higher Resolution, Same Latency](https://mlsanity.com/paper/2607.04443/) (2026)
- [Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts](https://mlsanity.com/paper/2609.17360/) (2026)
- [EvoAudio: Recursive Self-Improvement for Audio Understanding](https://mlsanity.com/paper/2609.27389/) (2026)
- [Thinking in Depth, Speaking Directly: Recurrent Latent Reasoning for Paralinguistically Grounded Spoken Dialogue](https://mlsanity.com/paper/2609.37818/) (2026)
- [Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time](https://mlsanity.com/paper/2605.01766/) (2026)
- [PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects](https://mlsanity.com/paper/2606.01016/) (2026)
- [HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence](https://mlsanity.com/paper/2608.26517/) (2026)
- [DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.03423/) (2026)
- [Never Stop Thinking: Continuous-Time Language Agents](https://mlsanity.com/paper/2609.17416/) (2026)
- [Full-Duplex Speech Models Take the Floor When Asked, Not When Needed](https://mlsanity.com/paper/2609.19596/) (2026)
- [Omni2Web: Benchmarking Audiovisual Website Development](https://mlsanity.com/paper/2609.23417/) (2026)
- [Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs](https://mlsanity.com/paper/2609.31193/) (2026)
- [SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.34247/) (2026)
- [Deep Pre-Alignment for VLMs](https://mlsanity.com/paper/2605.15300/) (2026)
- [video-SALMONN-R3: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding](https://mlsanity.com/paper/2606.24477/) (2026)

## Abstract (arXiv)

Recent progress in multimodal large language models (MLLMs) has brought AI capabilities from static offline data processing to real-time streaming interaction, yet they still remain far from human-level multimodal interaction. The key bottlenecks are no longer modality coverage or latency alone, but the interaction paradigm itself. First, perception and response are still separated into alternating phases, preventing models from incorporating new inputs for timely adjustment during generation. Second, most current models remain reactive, responding only to explicit user requests instead of acting proactively in the evolving multimodal environment. We present MiniCPM-o 4.5, our latest effort towards human-like multimodal interaction, which mitigates these gaps by real-time full-duplex omni-modal interaction. It can see, listen, and speak simultaneously in real-time, while also exhibiting proactive behaviors such as issuing reminders or comments based on its continuous understanding of the live scene. The key technique behind MiniCPM-o 4.5 is Omni-Flow, a unified streaming framework that aligns omni-modal inputs and outputs along a shared temporal axis. This formulation converts conventional turn-based interaction into a full-duplex, time-aligned process, enabling simultaneous perception and response and allowing proactive behavior to arise within the same framework. With a total of 9B parameters, MiniCPM-o 4.5 approaches Gemini 2.5 Flash in vision-language capabilities, delivering state-of-the-art open-source performance at its scale. It also surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and delivers better speech generation, with significantly higher computation efficiency. Driven by its efficient architecture design and inference optimization, the model can perform real-time full-duplex omni-modal interaction on edge devices with less than 12GB RAM cost.