# MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing arXiv: https://arxiv.org/abs/2502.21291 ยท HTML: https://mlsanity.com/paper/2502.21291/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 42 of its 67 references that are in the map. - First posted on arXiv 28 Feb 2025. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 42) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024) - [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022) - [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023) - [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022) - [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023) - [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023) - [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023) - [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024) - [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023) - [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024) - [AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea](https://mlsanity.com/paper/2411.15738/) (2024) - [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019) - [TokenPacker: Efficient Visual Projector for Multimodal LLM](https://mlsanity.com/paper/2407.02392/) (2024) - [Improving Diffusion Models for Authentic Virtual Try-on in the Wild](https://mlsanity.com/paper/2403.05139/) (2024) - [UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics](https://mlsanity.com/paper/2412.07774/) (2024) - [SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing](https://mlsanity.com/paper/2405.04007/) (2024) - [Inst-Inpaint: Instructing to Remove Objects with Diffusion Models](https://mlsanity.com/paper/2304.03246/) (2023) - [Instruct-Imagen: Image Generation with Multi-modal Instruction](https://mlsanity.com/paper/2401.01952/) (2024) - [Zero-shot Image Editing with Reference Imitation](https://mlsanity.com/paper/2406.07547/) (2024) - [ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer](https://mlsanity.com/paper/2410.00086/) (2024) - [One Diffusion to Generate Them All](https://mlsanity.com/paper/2411.16318/) (2024) ## Built on by (most cited first, 10 of 10) - [GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation](https://mlsanity.com/paper/2504.02782/) (2025) - [OminiControl2: Efficient Conditioning for Diffusion Transformers](https://mlsanity.com/paper/2503.08280/) (2025) - [Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning](https://mlsanity.com/paper/2507.01908/) (2025) - [Image Editing with Diffusion Models: A Survey](https://mlsanity.com/paper/2504.13226/) (2025) - [Dynamic-I2V: Exploring Image-to-Video Generation Models via Multimodal LLM](https://mlsanity.com/paper/2505.19901/) (2025) - [Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing](https://mlsanity.com/paper/2509.01986/) (2025) - [AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars](https://mlsanity.com/paper/2507.02419/) (2025) - [Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration](https://mlsanity.com/paper/2606.17296/) (2026) - [From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation](https://mlsanity.com/paper/2608.18076/) (2026) - [AlbumFill: Album-Guided Reasoning and Retrieval for Personalized Image Completion](https://mlsanity.com/paper/2605.02892/) (2026) ## Abstract (arXiv) Despite significant progress in diffusion-based image generation, subject-driven generation and instruction-based editing remain challenging. Existing methods typically treat them separately, struggling with limited high-quality data and poor generalization. However, both tasks require capturing complex visual variations while maintaining consistency between inputs and outputs. Inspired by this, we propose MIGE, a unified framework that standardizes task representations using multimodal instructions. It first treats subject-driven generation as creation on a blank canvas and instruction-based editing as modification of an existing image, establishing a shared input-output formulation, then introduces a novel multimodal encoder that maps free-form multimodal instructions into a unified vision-language space, integrating visual and semantic features through a feature fusion mechanism. This unification enables joint training of both tasks, providing two key advantages: (1) Cross-Task Enhancement: by leveraging shared visual and semantic representations, joint training improves instruction adherence and visual consistency in both subject-driven generation and instruction-based editing. (2) Generalization: learning in a unified format facilitates cross-task knowledge transfer, enabling MIGE to generalize to novel compositional tasks, including instruction-based subject-driven editing. Experiments show that MIGE excels in both subject-driven generation and instruction-based editing while setting a SOTA in the new task of instruction-based subject-driven editing. Code and model have been publicly available at https://github.com/Eureka-Maggie/MIGE.