Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
Not on the map. Why?
Abstract (arXiv)
In recent years, integrating multimodal understanding and generation into a single unified model has emerged as a promising paradigm. While this approach achieves strong results in text-to-image (T2I) generation, it still struggles with precise image editing. We attribute this limitation to an imbalanced division of responsibilities. The understanding module primarily functions as a translator that encodes user instructions into semantic conditions, while the generation module must simultaneously act as designer and painter, inferring the original layout, identifying the target editing region, and rendering the new content. This imbalance is counterintuitive because the understanding module is typically trained with several times more data on complex reasoning tasks than the generation module. To address this issue, we introduce Draw-In-Mind (DIM), a dataset comprising two complementary subsets: (i) DIM-T2I, containing 14M long-context image-text pairs to enhance complex instruction comprehension; and (ii) DIM-Edit, consisting of 233K chain-of-thought imaginations generated by GPT-4o, serving as explicit design blueprints for image edits. We connect a frozen Qwen2.5-VL-3B with a trainable SANA1.5-1.6B via a lightweight two-layer MLP, and train it on the proposed DIM dataset, resulting in DIM-4.6B-T2I/Edit. Despite its modest parameter scale, DIM-4.6B-Edit achieves SOTA or competitive performance on the ImgEdit and GEdit-Bench benchmarks, outperforming much larger models such as UniWorld-V1 and Step1X-Edit. These findings demonstrate that explicitly assigning the design responsibility to the understanding module provides significant benefits for image editing. Our dataset and models are available at https://github.com/showlab/DIM.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 61 of its 67 references that are in the map. First posted on arXiv 2 Sep 2025.
Builds on
61 of 67 references.
References · 61
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Visual Instruction Tuningcited by 7,869
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
38 more
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2019Towards VQA Models That Can Readcited by 1,718
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2016A Diagram Is Worth A Dozen Imagescited by 804
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
- 2024OmniGen: Unified Image Generationcited by 394
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
- 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
- 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
- 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
- 2025Transfer between Modalities with MetaQueriescited by 236
- 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
- 2024Taming Rectified Flow for Inversion and Editingcited by 204
- 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
- 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
- 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
- 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
- 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generationcited by 1
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editingcited by 1
- 2026InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoningcited by 0
- 2026Attention-Scoped Guidance: Training-Free Spatial Control for Image Editingcited by 0
- 2026Attention-Scoped Guidance: Training-Free Spatial Control for Image Editing28 Sep 2026
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing4 Sep 2026
- 2026Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation19 May 2026
- 2026InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning2 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing
- 2026Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Models
- 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing
- 2025Factuality Matters: When Image Generation and Editing Meet Structured Visuals
- 2024EVLM: Self-Reflective Multimodal Reasoning for Cross-Dimensional Visual Editing
- 2026UniReason 1.0: A Unified Reasoning Framework for World Knowledge Aligned Image Generation and Editing
- 2026Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editing
- 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs