# UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark

arXiv: https://arxiv.org/abs/2603.05075 · HTML: https://mlsanity.com/paper/2603.05075/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 46 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 30 of 44)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [Emu3.5: Native Multimodal Models are World Learners](https://mlsanity.com/paper/2510.26583/) (2025)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://mlsanity.com/paper/2401.10208/) (2024)
- [Ming-Omni: A Unified Multimodal Model for Perception and Generation](https://mlsanity.com/paper/2506.09344/) (2025)
- [CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation](https://mlsanity.com/paper/2311.18775/) (2023)
- [4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities](https://mlsanity.com/paper/2406.09406/) (2024)
- [OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation](https://mlsanity.com/paper/2509.03498/) (2025)
- [Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing](https://mlsanity.com/paper/2412.19806/) (2024)
- [X-VILA: Cross-Modality Alignment for Large Language Model](https://mlsanity.com/paper/2405.19335/) (2024)
- [OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows](https://mlsanity.com/paper/2412.01169/) (2024)
- [MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2410.10139/) (2024)

## Built on by (most cited first, 11 of 11)

- [ParseFixer: An Agentic Framework for Document Parsing via Selective Multimodal Correction](https://mlsanity.com/paper/2606.11977/) (2026)
- [ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement](https://mlsanity.com/paper/2606.10640/) (2026)
- [SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection](https://mlsanity.com/paper/2607.03069/) (2026)
- [UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound](https://mlsanity.com/paper/2609.30928/) (2026)
- [Enhancing Biomedical Named Entity Recognition via Multiple Programming Languages Instruction Tuning and Ensemble Method](https://mlsanity.com/paper/2610.02949/) (2026)
- [CALMRec: Causally Aligned Language Memory for Long-Horizon Recommendation](https://mlsanity.com/paper/2607.23647/) (2026)
- [CRAFTS: Collaborative Role-Adaptive Fine-Tuning of LLM Agents for Chemical Process Simulation](https://mlsanity.com/paper/2608.01369/) (2026)
- [T2exture: Sparsely Perturbed Thermal-to-Texture Imaging](https://mlsanity.com/paper/2608.02192/) (2026)
- [StructReward: Efficient Structured Process Rewards for Self-Correcting Multimodal Reasoning](https://mlsanity.com/paper/2608.08326/) (2026)
- [Omni-IO Skills: Harnessing Your Agent Omni-Native](https://mlsanity.com/paper/2609.31847/) (2026)
- [What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation](https://mlsanity.com/paper/2609.37317/) (2026)

## Abstract (arXiv)

In real-world multimodal applications, systems usually need to comprehend arbitrarily combined and interleaved multimodal inputs from users, while also generating outputs in any interleaved multimedia form. This capability defines the goal of any-to-any interleaved multimodal learning under a unified paradigm of understanding and generation, posing new challenges and opportunities for advancing Multimodal Large Language Models (MLLMs). To foster and benchmark this capability, this paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset. UniM contains 31K high-quality instances across 30 domains and 7 representative modalities: text, image, audio, video, document, code, and 3D, each requiring multiple intertwined reasoning and generation capabilities. We further introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness & Generation Quality, Response Structure Integrity, and Interleaved Coherence. In addition, we propose UniMA, an agentic baseline model equipped with traceable reasoning for structured interleaved generation. Comprehensive experiments demonstrate the difficulty of UniM and highlight key challenges and directions for advancing unified any-to-any multimodal intelligence. The project page is https://any2any-mllm.github.io/unim.