# DreamOmni: Unified Image Generation and Editing

arXiv: https://arxiv.org/abs/2412.17098 · HTML: https://mlsanity.com/paper/2412.17098/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 64 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 30 of 59)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)

## Built on by (most cited first, 30 of 30)

- [EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer](https://mlsanity.com/paper/2503.07027/) (2025)
- [DreamOmni2: Multimodal Instruction-based Editing and Generation](https://mlsanity.com/paper/2510.06679/) (2025)
- [OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning](https://mlsanity.com/paper/2603.24458/) (2026)
- [VINCIE: Unlocking In-context Image Editing from Video](https://mlsanity.com/paper/2506.10941/) (2025)
- [DreamVE: Unified Instruction-based Image and Video Editing](https://mlsanity.com/paper/2508.06080/) (2025)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [SpatialEdit: Benchmarking Fine-Grained Image Spatial Editing](https://mlsanity.com/paper/2604.04911/) (2026)
- [Omni2: Unifying Omnidirectional Image Generation and Editing in an Omni Model](https://mlsanity.com/paper/2504.11379/) (2025)
- [GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing](https://mlsanity.com/paper/2603.12264/) (2026)
- [iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generation](https://mlsanity.com/paper/2511.20635/) (2025)
- [EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection](https://mlsanity.com/paper/2510.13652/) (2025)
- [The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment](https://mlsanity.com/paper/2511.20614/) (2025)
- [VDOT: Efficient Unified Video Creation via Optimal Transport Distillation](https://mlsanity.com/paper/2512.06802/) (2025)
- [TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models](https://mlsanity.com/paper/2601.08011/) (2026)
- [LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editing](https://mlsanity.com/paper/2508.03144/) (2025)
- [CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generation](https://mlsanity.com/paper/2602.22150/) (2026)
- [DreamOmni3: Scribble-based Editing and Generation](https://mlsanity.com/paper/2512.22525/) (2025)
- [VIBE: Visual Instruction Based Editor](https://mlsanity.com/paper/2601.02242/) (2026)
- [From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation](https://mlsanity.com/paper/2608.18076/) (2026)
- [Geometry-Editable and Appearance-Preserving Object Compositon](https://mlsanity.com/paper/2505.20914/) (2025)
- [BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models](https://mlsanity.com/paper/2605.28067/) (2026)
- [VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation](https://mlsanity.com/paper/2608.13613/) (2026)
- [RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generation](https://mlsanity.com/paper/2507.02792/) (2025)
- [Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation](https://mlsanity.com/paper/2605.26111/) (2026)
- [In-context Region-based Drag: Drag Any Region to Any Shape](https://mlsanity.com/paper/2606.25907/) (2026)
- [UniMoFlow: Grounding Instruction-Driven 3D Human Motion Editing in Generation](https://mlsanity.com/paper/2608.09143/) (2026)
- [Measured Sliders: Learning Continuous Controls from Differentiable Image Measurements](https://mlsanity.com/paper/2609.05234/) (2026)
- [KwaiMind Technical Report](https://mlsanity.com/paper/2609.26375/) (2026)
- [RefAdapt-DiT: Adaptive Joint Attention for Reference-Conditioned Diffusion Transformers](https://mlsanity.com/paper/2609.32415/) (2026)
- [UMAS3D: Unified Mask-Free 3D Editing with Region-Adaptive Source Preservation](https://mlsanity.com/paper/2610.15533/) (2026)

## Abstract (arXiv)

Currently, the success of large language models (LLMs) illustrates that a unified multitasking approach can significantly enhance model usability, streamline deployment, and foster synergistic benefits across different tasks. However, in computer vision, while text-to-image (T2I) models have significantly improved generation quality through scaling up, their framework design did not initially consider how to unify with downstream tasks, such as various types of editing. To address this, we introduce DreamOmni, a unified model for image generation and editing. We begin by analyzing existing frameworks and the requirements of downstream tasks, proposing a unified framework that integrates both T2I models and various editing tasks. Furthermore, another key challenge is the efficient creation of high-quality editing data, particularly for instruction-based and drag-based editing. To this end, we develop a synthetic data pipeline using sticker-like elements to synthesize accurate, high-quality datasets efficiently, which enables editing data scaling up for unified model training. For training, DreamOmni jointly trains T2I generation and downstream tasks. T2I training enhances the model's understanding of specific concepts and improves generation quality, while editing training helps the model grasp the nuances of the editing task. This collaboration significantly boosts editing performance. Extensive experiments confirm the effectiveness of DreamOmni. The code and model will be released.