mlsanityData to 7 Oct 2026

UltraEdit: Instruction-based Fine-Grained Image Editing at Scale

arXiv 2407.05282 ↗First posted 7 Jul 2024cs.CVcited by 266

Not on the map. Why?

Cited by 64 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents UltraEdit, a large-scale (approximately 4 million editing samples), automatically generated dataset for instruction-based image editing. Our key idea is to address the drawbacks in existing image editing datasets like InstructPix2Pix and MagicBrush, and provide a systematic approach to producing massive and high-quality image editing samples. UltraEdit offers several distinct advantages: 1) It features a broader range of editing instructions by leveraging the creativity of large language models (LLMs) alongside in-context editing examples from human raters; 2) Its data sources are based on real images, including photographs and artworks, which provide greater diversity and reduced bias compared to datasets solely generated by text-to-image models; 3) It also supports region-based editing, enhanced by high-quality, automatically produced region annotations. Our experiments show that canonical diffusion-based editing baselines trained on UltraEdit set new records on MagicBrush and Emu-Edit benchmarks. Our analysis further confirms the crucial role of real image anchors and region-based editing data. The dataset, code, and models can be found in https://ultra-editing.github.io.

Key facts. Cited by 266 papers in the map (data through 7 Oct 2026). Builds on 53 of its 68 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

53 of 68 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Segment Anythingcited by 8,158
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
38 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2018Datasheets for Datasetscited by 1,371
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  13. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  14. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  15. 2018nocaps: novel object captioning at scalecited by 444
  16. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  17. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  18. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  19. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  20. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  21. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  22. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  23. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  24. 2019Connecting Vision and Language with Localized Narrativescited by 225
  25. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  26. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  27. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  28. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  29. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 191
  30. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  31. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  32. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 168
  33. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  34. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 149
  35. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  36. 2023Social Biases through the Text-to-Image Generation Lenscited by 126
  37. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  38. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 135; 2026: 111.

266 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2025VACE: All-in-One Video Creation and Editingcited by 467
  4. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  6. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  7. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  8. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  9. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  10. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  11. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  12. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
38 more
  1. 2024Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advancescited by 105
  2. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  3. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  4. 2025LongCat-Image Technical Reportcited by 92
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  6. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  7. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  8. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  9. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  10. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  11. 2025GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Datasetcited by 68
  12. 2025Ovis-U1 Technical Reportcited by 65
  13. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  14. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  15. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  16. 2024SeedEdit: Align Image Re-Generation to Image Editingcited by 60
  17. 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editingcited by 59
  18. 2025Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editingcited by 58
  19. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  20. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  21. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  22. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  23. 2025SeedEdit 3.0: Fast and High-Quality Generative Image Editingcited by 52
  24. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  25. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  26. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 50
  27. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  28. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  29. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  30. 2025UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Modelscited by 38
  31. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 35
  32. 2024Generative Image Layer Decomposition with Visual Effectscited by 33
  33. 2025NoHumansRequired: Autonomous High-Quality Image Editing Triplet Miningcited by 33
  34. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  35. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 31
  36. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  37. 2025FlowAlign: Trajectory-Regularized, Inversion-Free Flow-based Image Editingcited by 31
  38. 2026FireRed-Image-Edit-1.0 Technical Reportcited by 31

50 most cited of 266 All 266 (JSON)