# Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space

arXiv: https://arxiv.org/abs/2504.21356 · HTML: https://mlsanity.com/paper/2504.21356/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea](https://mlsanity.com/paper/2411.15738/) (2024)
- [JourneyDB: A Benchmark for Generative Image Understanding](https://mlsanity.com/paper/2307.00716/) (2023)
- [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2404.09990/) (2024)
- [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023)
- [OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision](https://mlsanity.com/paper/2411.07199/) (2024)
- [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](https://mlsanity.com/paper/2504.17432/) (2025)
- [InfiniteYou: Flexible Photo Recrafting While Preserving Your Identity](https://mlsanity.com/paper/2503.16418/) (2025)
- [EliGen: Entity-Level Controlled Image Generation with Regional Attention](https://mlsanity.com/paper/2501.01097/) (2025)

## Built on by (most cited first, 9 of 9)

- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation](https://mlsanity.com/paper/2505.24787/) (2025)
- [UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis](https://mlsanity.com/paper/2510.15710/) (2025)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization](https://mlsanity.com/paper/2511.05616/) (2025)
- [InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback](https://mlsanity.com/paper/2505.23950/) (2025)
- [FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration](https://mlsanity.com/paper/2511.14099/) (2025)
- [UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy](https://mlsanity.com/paper/2603.24690/) (2026)
- [ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation](https://mlsanity.com/paper/2606.23835/) (2026)

## Abstract (arXiv)

Unified multimodal generative models aim to integrate image understanding and generation abilities, offering significant advantages in harnessing multimodal corpora, particularly interleaved text-image data. However, existing unified models exhibit limitations in image synthesis quality, autoregressive error accumulation, and image editing capability. In this work, we propose Nexus-Gen, a novel architecture that unifies image understanding, generation, and editing tasks in a shared image embedding space. This shared space serves as a bridge for the autoregressive and diffusion models, which seamlessly integrates their complementary strengths in cross-modal modeling. To mitigate the severe error accumulation during autoregressive embedding prediction, we propose a novel prefilled autoregression strategy that aligns training-inference dynamics by prefilling input sequences with learnable embeddings. After multi-stage and multi-task training on our constructed large-scale dataset with 26.3 million samples, Nexus-Gen achieves state-of-the-art performance on the evaluation benchmarks spanning image understanding, generation and editing tasks. All models, datasets, and source codes are released in https://github.com/modelscope/Nexus-Gen to facilitate further advancements across the field.