mlsanityData to 14 Oct 2026

Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing

arXiv 2509.01986 ↗First posted 2 Sep 2025cs.CV cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, integrating multimodal understanding and generation into a single unified model has emerged as a promising paradigm. While this approach achieves strong results in text-to-image (T2I) generation, it still struggles with precise image editing. We attribute this limitation to an imbalanced division of responsibilities. The understanding module primarily functions as a translator that encodes user instructions into semantic conditions, while the generation module must simultaneously act as designer and painter, inferring the original layout, identifying the target editing region, and rendering the new content. This imbalance is counterintuitive because the understanding module is typically trained with several times more data on complex reasoning tasks than the generation module. To address this issue, we introduce Draw-In-Mind (DIM), a dataset comprising two complementary subsets: (i) DIM-T2I, containing 14M long-context image-text pairs to enhance complex instruction comprehension; and (ii) DIM-Edit, consisting of 233K chain-of-thought imaginations generated by GPT-4o, serving as explicit design blueprints for image edits. We connect a frozen Qwen2.5-VL-3B with a trainable SANA1.5-1.6B via a lightweight two-layer MLP, and train it on the proposed DIM dataset, resulting in DIM-4.6B-T2I/Edit. Despite its modest parameter scale, DIM-4.6B-Edit achieves SOTA or competitive performance on the ImgEdit and GEdit-Bench benchmarks, outperforming much larger models such as UniWorld-V1 and Step1X-Edit. These findings demonstrate that explicitly assigning the design responsibility to the understanding module provides significant benefits for image editing. Our dataset and models are available at https://github.com/showlab/DIM.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 61 of its 67 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

61 of 67 references.

References · 61

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  13. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  15. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  16. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  17. 2016A Diagram Is Worth A Dozen Imagescited by 804
  18. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  19. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  20. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  21. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  22. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  23. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  24. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  25. 2024OmniGen: Unified Image Generationcited by 394
  26. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  27. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  28. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  29. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  30. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  31. 2025Transfer between Modalities with MetaQueriescited by 236
  32. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  33. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  34. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
  35. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
  36. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  37. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  38. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing
  2. 2026Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Models
  3. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing
  4. 2025Factuality Matters: When Image Generation and Editing Meet Structured Visuals
  5. 2024EVLM: Self-Reflective Multimodal Reasoning for Cross-Dimensional Visual Editing
  6. 2026UniReason 1.0: A Unified Reasoning Framework for World Knowledge Aligned Image Generation and Editing
  7. 2026Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editing
  8. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs