# DreamOmni2: Multimodal Instruction-based Editing and Generation

arXiv: https://arxiv.org/abs/2510.06679 · HTML: https://mlsanity.com/paper/2510.06679/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 30 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 27 of 27)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [OmniGen2: Towards Instruction-Aligned Multimodal Generation](https://mlsanity.com/paper/2506.18871/) (2025)
- [In-Context LoRA for Diffusion Transformers](https://mlsanity.com/paper/2410.23775/) (2024)
- [Less-to-More Generalization: Unlocking More Controllability by In-Context Generation](https://mlsanity.com/paper/2504.02160/) (2025)
- [OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision](https://mlsanity.com/paper/2411.07199/) (2024)
- [SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing](https://mlsanity.com/paper/2405.04007/) (2024)
- [DreamO: A Unified Framework for Image Customization](https://mlsanity.com/paper/2504.16915/) (2025)
- [XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation](https://mlsanity.com/paper/2506.21416/) (2025)
- [DreamOmni: Unified Image Generation and Editing](https://mlsanity.com/paper/2412.17098/) (2024)
- [LLMGA: Multimodal Large Language Model based Generation Assistant](https://mlsanity.com/paper/2311.16500/) (2023)
- [DreamVE: Unified Instruction-based Image and Video Editing](https://mlsanity.com/paper/2508.06080/) (2025)

## Built on by (most cited first, 30 of 38)

- [FireRed-Image-Edit-1.0 Technical Report](https://mlsanity.com/paper/2602.13344/) (2026)
- [HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer](https://mlsanity.com/paper/2605.11061/) (2026)
- [JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2605.04128/) (2026)
- [WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing](https://mlsanity.com/paper/2512.00387/) (2025)
- [Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editing](https://mlsanity.com/paper/2601.05124/) (2026)
- [I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models](https://mlsanity.com/paper/2512.04660/) (2025)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [TAG-MoE: Task-Aware Gating for Unified Generative Mixture-of-Experts](https://mlsanity.com/paper/2601.08881/) (2026)
- [MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models](https://mlsanity.com/paper/2602.19497/) (2026)
- [OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation](https://mlsanity.com/paper/2512.08294/) (2025)
- [Omni-Attribute: Open-vocabulary Attribute Encoder for Visual Concept Personalization](https://mlsanity.com/paper/2512.10955/) (2025)
- [GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?](https://mlsanity.com/paper/2602.06013/) (2026)
- [Garments2Look: A Multi-Reference Dataset for High-Fidelity Outfit-Level Virtual Try-On with Clothing and Accessories](https://mlsanity.com/paper/2603.14153/) (2026)
- [EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editing](https://mlsanity.com/paper/2603.14916/) (2026)
- [Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing](https://mlsanity.com/paper/2606.05172/) (2026)
- [Scaling Multi-Reference Image Generation with Dynamic Reward Optimization](https://mlsanity.com/paper/2606.26947/) (2026)
- [Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing](https://mlsanity.com/paper/2607.19064/) (2026)
- [DreamOmni3: Scribble-based Editing and Generation](https://mlsanity.com/paper/2512.22525/) (2025)
- [AIM-Bench: Benchmarking and Improving Affective Image Manipulation via Fine-Grained Hierarchical Control](https://mlsanity.com/paper/2604.10454/) (2026)
- [Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation](https://mlsanity.com/paper/2605.12305/) (2026)
- [Exploring the Performance Frontier of Compact Unified Image Generation Models](https://mlsanity.com/paper/2608.20334/) (2026)
- [Towards Generalized Multi-Image Editing for Unified Multimodal Models](https://mlsanity.com/paper/2601.05572/) (2026)
- [VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On](https://mlsanity.com/paper/2603.11734/) (2026)
- [MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisal](https://mlsanity.com/paper/2603.15020/) (2026)
- [OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework](https://mlsanity.com/paper/2603.19643/) (2026)
- [OmniColor: A Unified Framework for Multi-modal Lineart Colorization](https://mlsanity.com/paper/2603.27531/) (2026)
- [FED-Bench: A Cross-Granular Benchmark for Disentangled Evaluation of Facial Expression Editing](https://mlsanity.com/paper/2603.29697/) (2026)
- [MooD: Perception-Enhanced Efficient Affective Image Editing via Continuous Valence-Arousal Modeling](https://mlsanity.com/paper/2605.02521/) (2026)
- [InsHuman: Towards Natural and Identity-Preserving Human Insertion](https://mlsanity.com/paper/2605.07402/) (2026)
- [EditRefiner: A Human-Aligned Agentic Framework for Image Editing Refinement](https://mlsanity.com/paper/2605.07457/) (2026)

## Abstract (arXiv)

Recent advancements in instruction-based image editing and subject-driven generation have garnered significant attention, yet both tasks still face limitations in meeting practical user needs. Instruction-based editing relies solely on language instructions, which often fail to capture specific editing details, making reference images necessary. Meanwhile, subject-driven generation is limited to combining concrete objects or people, overlooking broader, abstract concepts. To address these challenges, we propose two novel tasks: multimodal instruction-based editing and generation. These tasks support both text and image instructions and extend the scope to include both concrete and abstract concepts, greatly enhancing their practical applications. We introduce DreamOmni2, tackling two primary challenges: data creation and model framework design. Our data synthesis pipeline consists of three steps: (1) using a feature mixing method to create extraction data for both abstract and concrete concepts, (2) generating multimodal instruction-based editing training data using the editing and extraction models, and (3) further applying the extraction model to create training data for multimodal instruction-based editing. For the framework, to handle multi-image input, we propose an index encoding and position encoding shift scheme, which helps the model distinguish images and avoid pixel confusion. Additionally, we introduce joint training with the VLM and our generation/editing model to better process complex instructions. In addition, we have proposed comprehensive benchmarks for these two new tasks to drive their development. Experiments show that DreamOmni2 has achieved impressive results. Models and codes will be released.