# Context Unrolling in Omni Models

arXiv: https://arxiv.org/abs/2604.21921 · HTML: https://mlsanity.com/paper/2604.21921/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 47 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Builds on (most cited first, 30 of 44)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer](https://mlsanity.com/paper/2511.22699/) (2025)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)

## Built on by (most cited first, 8 of 8)

- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [Bernini: Latent Semantic Planning for Video Diffusion](https://mlsanity.com/paper/2605.22344/) (2026)
- [Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers](https://mlsanity.com/paper/2607.21594/) (2026)
- [Representation Forcing for Bottleneck-Free Unified Multimodal Models](https://mlsanity.com/paper/2605.31604/) (2026)
- [Modality Forcing for Scalable Spatial Generation](https://mlsanity.com/paper/2606.13676/) (2026)
- [Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE](https://mlsanity.com/paper/2605.02641/) (2026)
- [Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes](https://mlsanity.com/paper/2608.05000/) (2026)
- [MoRoute: Dynamic Routing for In-Context Multimodal Video Generation](https://mlsanity.com/paper/2607.29545/) (2026)

## Abstract (arXiv)

We present Omni, a unified multimodal model natively trained on diverse modalities, including text, images, videos, 3D geometry, and hidden representations. We find that such training enables Context Unrolling, where the model explicitly reasons across multiple modal representations before producing predictions. This process enables the model to aggregate complementary information across heterogeneous modalities, facilitating a more faithful approximation of the shared multimodal knowledge manifold and improving downstream reasoning fidelity. As a result, Omni achieves strong performance on both multimodal generation and understanding benchmarks, while demonstrating advanced multimodal reasoning capabilities, including in-context generation of text, image, video, and 3D geometry.