# UniFusion: Vision-Language Model as Unified Encoder in Image Generation arXiv: https://arxiv.org/abs/2510.12789 ยท HTML: https://mlsanity.com/paper/2510.12789/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 31 of its 44 references that are in the map. - First posted on arXiv 14 Oct 2025. ## Placed in - Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/) - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 31) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025) - [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024) - [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019) - [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019) - [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025) - [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025) - [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023) - [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025) - [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025) - [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024) - [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024) - [Seedream 4.0: Toward Next-generation Multimodal Image Generation](https://mlsanity.com/paper/2509.20427/) (2025) - [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023) - [Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning](https://mlsanity.com/paper/2309.02591/) (2023) - [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://mlsanity.com/paper/2406.13743/) (2024) - [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020) - [Lumina-Image 2.0: A Unified and Efficient Image Generative Framework](https://mlsanity.com/paper/2503.21758/) (2025) - [SeedEdit 3.0: Fast and High-Quality Generative Image Editing](https://mlsanity.com/paper/2506.05083/) (2025) - [Intrinsic Probing through Dimension Selection](https://mlsanity.com/paper/2010.02812/) (2020) - [Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models](https://mlsanity.com/paper/2406.11831/) (2024) - [AltDiffusion: A Multilingual Text-to-Image Diffusion Model](https://mlsanity.com/paper/2308.09991/) (2023) - [GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation](https://mlsanity.com/paper/2303.10056/) (2023) - [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022) - [TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training](https://mlsanity.com/paper/2501.04765/) (2025) - [Residual Stream Analysis with Multi-Layer SAEs](https://mlsanity.com/paper/2409.04185/) (2024) - [MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation](https://mlsanity.com/paper/2305.15296/) (2023) ## Built on by (most cited first, 8 of 8) - [DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing](https://mlsanity.com/paper/2602.12205/) (2026) - [MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation](https://mlsanity.com/paper/2511.18262/) (2025) - [Exploring the Performance Frontier of Compact Unified Image Generation Models](https://mlsanity.com/paper/2608.20334/) (2026) - [ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction](https://mlsanity.com/paper/2512.05422/) (2025) - [LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation](https://mlsanity.com/paper/2604.00829/) (2026) - [High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding](https://mlsanity.com/paper/2603.13389/) (2026) - [Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation](https://mlsanity.com/paper/2605.26111/) (2026) - [Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning](https://mlsanity.com/paper/2610.09450/) (2026) ## Abstract (arXiv) Although recent advances in visual generation have been remarkable, most existing architectures still depend on distinct encoders for images and text. This separation constrains diffusion models' ability to perform cross-modal reasoning and knowledge transfer. Prior attempts to bridge this gap often use the last layer information from VLM, employ multiple visual encoders, or train large unified models jointly for text and image generation, which demands substantial computational resources and large-scale data, limiting its accessibility.We present UniFusion, a diffusion-based generative model conditioned on a frozen large vision-language model (VLM) that serves as a unified multimodal encoder. At the core of UniFusion is the Layerwise Attention Pooling (LAP) mechanism that extracts both high level semantics and low level details from text and visual tokens of a frozen VLM to condition a diffusion generative model. We demonstrate that LAP outperforms other shallow fusion architectures on text-image alignment for generation and faithful transfer of visual information from VLM to the diffusion model which is key for editing. We propose VLM-Enabled Rewriting Injection with Flexibile Inference (VERIFI), which conditions a diffusion transformer (DiT) only on the text tokens generated by the VLM during in-model prompt rewriting. VERIFI combines the alignment of the conditioning distribution with the VLM's reasoning capabilities for increased capabilities and flexibility at inference. In addition, finetuning on editing task not only improves text-image alignment for generation, indicative of cross-modality knowledge transfer, but also exhibits tremendous generalization capabilities. Our model when trained on single image editing, zero-shot generalizes to multiple image references further motivating the unified encoder design of UniFusion.