# Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation

arXiv: https://arxiv.org/abs/2610.17080 · HTML: https://mlsanity.com/paper/2610.17080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 30 of 33)

- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024)
- [Rolling Forcing: Autoregressive Long Video Diffusion in Real Time](https://mlsanity.com/paper/2509.25161/) (2025)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [Self-Forcing++: Towards Minute-Scale High-Quality Video Generation](https://mlsanity.com/paper/2510.02283/) (2025)
- [GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing](https://mlsanity.com/paper/2407.05600/) (2024)
- [Emu3.5: Native Multimodal Models are World Learners](https://mlsanity.com/paper/2510.26583/) (2025)
- [SEED-Story: Multimodal Long Story Generation with Large Language Model](https://mlsanity.com/paper/2407.08683/) (2024)
- [Thinking with Generated Images](https://mlsanity.com/paper/2505.22525/) (2025)
- [Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision](https://mlsanity.com/paper/2508.05606/) (2025)
- [Qwen-Image-2.0 Technical Report](https://mlsanity.com/paper/2605.10730/) (2026)
- [HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer](https://mlsanity.com/paper/2605.11061/) (2026)
- [Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation](https://mlsanity.com/paper/2606.26907/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training](https://mlsanity.com/paper/2603.25706/) (2026)
- [Loom: Diffusion-Transformer for Interleaved Generation](https://mlsanity.com/paper/2512.18254/) (2025)
- [TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding](https://mlsanity.com/paper/2606.08464/) (2026)
- [ChatUMM: Robust Context Tracking for Conversational Interleaved Generation](https://mlsanity.com/paper/2602.06442/) (2026)

## Abstract (arXiv)

Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.