mlsanityData to 7 Oct 2026

GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing

arXiv 2503.10639 ↗First posted 13 Mar 2025cs.CVcited by 103

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Current image generation and editing methods primarily process textual prompts as direct inputs without reasoning about visual composition and explicit operations. We present Generation Chain-of-Thought (GoT), a novel paradigm that enables generation and editing through an explicit language reasoning process before outputting images. This approach transforms conventional text-to-image generation and editing into a reasoning-guided framework that analyzes semantic relationships and spatial arrangements. We define the formulation of GoT and construct large-scale GoT datasets containing over 9M samples with detailed reasoning chains capturing semantic-spatial relationships. To leverage the advantages of GoT, we implement a unified framework that integrates Qwen2.5-VL for reasoning chain generation with an end-to-end diffusion model enhanced by our novel Semantic-Spatial Guidance Module. Experiments show our GoT framework achieves excellent performance on both generation and editing tasks, with significant improvements over baselines. Additionally, our approach enables interactive visual generation, allowing users to explicitly modify reasoning steps for precise image adjustments. GoT pioneers a new direction for reasoning-driven visual generation and editing, producing images that better align with human intent. To facilitate future research, we make our datasets, code, and pretrained models publicly available at https://github.com/rongyaofang/GoT.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 54 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

49 of 54 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2025Qwen2.5-VL Technical Reportcited by 5,418
  12. 2024Qwen2.5 Technical Reportcited by 4,713
37 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  13. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  14. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  15. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  16. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  17. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  18. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  19. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  20. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  21. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  22. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  23. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  24. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  25. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  26. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  27. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  28. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  29. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  30. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  31. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  32. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  33. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 75
  34. 2023InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalistscited by 30
  35. 2024PUMA: Empowering Unified MLLM with Multi-granular Visual Generationcited by 20
  36. 2023InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generationcited by 14
  37. 2024FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image Editingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 59; 2026: 44.

103 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  2. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  5. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  6. 2025Thinking with Generated Imagescited by 71
  7. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  8. 2025Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizablecited by 58
  9. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  10. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 51
  11. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  12. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
38 more
  1. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 40
  2. 2025T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generationcited by 38
  3. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 37
  4. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  5. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  6. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generationcited by 30
  7. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 30
  8. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 27
  9. 2025ReasonEdit: Towards Reasoning-Enhanced Image Editing Modelscited by 24
  10. 2025CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Imagescited by 22
  11. 2025WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generationcited by 20
  12. 2025ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependenciescited by 20
  13. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  14. 2025Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generationcited by 17
  15. 2025Factuality Matters: When Image Generation and Editing Meet Structured Visualscited by 16
  16. 2026GEMS: Agent-Native Multimodal Generation with Memory and Skillscited by 16
  17. 2025ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedbackcited by 15
  18. 2025Reinforcing Video Reasoning Segmentation to Think Before It Segmentscited by 15
  19. 2025Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generationcited by 14
  20. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  21. 2025CompBench: Benchmarking Complex Instruction-guided Image Editingcited by 14
  22. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generationcited by 14
  23. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  24. 2025EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editingcited by 12
  25. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editingcited by 12
  26. 2026Demystifying Video Reasoningcited by 12
  27. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  28. 2026UniReason 1.0: A Unified Reasoning Framework for World Knowledge Aligned Image Generation and Editingcited by 11
  29. 2025Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructionscited by 10
  30. 2026CoF-T2I: Video Models as Pure Visual Reasoners for Text-to-Image Generationcited by 9
  31. 2025Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learningcited by 8
  32. 2025ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoningcited by 8
  33. 2025MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoningcited by 8
  34. 2025MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generationcited by 8
  35. 2026UniT: Unified Multimodal Chain-of-Thought Test-time Scalingcited by 8
  36. 2025I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Modelscited by 7
  37. 2025R-Genie: Reasoning-Guided Generative Image Editingcited by 6
  38. 2025Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regionscited by 6

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generation
  2. 2025GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
  3. 2026Spatial Chain-of-Thought: Bridging Understanding and Generation Models for Spatial Reasoning Generation
  4. 2026Reasoning with Image Generation
  5. 2025Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
  6. 2026ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis
  7. 2025ThinkGen: Generalized Thinking for Visual Generation
  8. 2026ETCHR: Editing To Clarify and Harness Reasoning