# OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing

arXiv: https://arxiv.org/abs/2508.04361 · HTML: https://mlsanity.com/paper/2508.04361/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 16 of 16)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://mlsanity.com/paper/2501.01957/) (2025)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments](https://mlsanity.com/paper/2402.16499/) (2024)

## Built on by (most cited first, 4 of 4)

- [Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning](https://mlsanity.com/paper/2509.23744/) (2025)
- [Sandboxed Coding Agents are Competitive Omni-modal Task Solvers](https://mlsanity.com/paper/2606.00579/) (2026)
- [Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time](https://mlsanity.com/paper/2604.08340/) (2026)
- [MMTB: Evaluating Terminal Agents on Multimedia-File Tasks](https://mlsanity.com/paper/2605.10966/) (2026)

## Abstract (arXiv)

While generalist foundation models like Gemini and GPT-4o demonstrate impressive multi-modal competence, existing evaluations fail to test their intelligence in dynamic, interactive worlds. Static benchmarks lack agency, while interactive benchmarks suffer from a severe modal bottleneck, typically ignoring crucial auditory and temporal cues. To bridge this evaluation chasm, we introduce OmniPlay, a diagnostic benchmark designed not just to evaluate, but to probe the fusion and reasoning capabilities of agentic models across the full sensory spectrum. Built on a core philosophy of modality interdependence, OmniPlay comprises a suite of five game environments that systematically create scenarios of both synergy and conflict, forcing agents to perform genuine cross-modal reasoning. Our comprehensive evaluation of six leading omni-modal models reveals a critical dichotomy: they exhibit superhuman performance on high-fidelity memory tasks but suffer from systemic failures in challenges requiring robust reasoning and strategic planning. We demonstrate that this fragility stems from brittle fusion mechanisms, which lead to catastrophic performance degradation under modality conflict and uncover a counter-intuitive "less is more" paradox, where removing sensory information can paradoxically improve performance. Our findings suggest that the path toward robust AGI requires a research focus beyond scaling to explicitly address synergistic fusion. Our platform is available for anonymous review at https://github.com/fuqingbie/omni-game-benchmark.