# Fine-grained Image Editing by Pixel-wise Guidance Using Diffusion Models

arXiv: https://arxiv.org/abs/2212.02024 · HTML: https://mlsanity.com/paper/2212.02024/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 54 of its 62 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 54)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)

## Built on by (most cited first, 18 of 18)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2307.16687/) (2023)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)
- [Diffusion-based Image Translation with Label Guidance for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2308.12350/) (2023)
- [Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners](https://mlsanity.com/paper/2305.10722/) (2023)
- [On the Limitations of Vision-Language Models in Understanding Image Transforms](https://mlsanity.com/paper/2503.09837/) (2025)
- [Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning](https://mlsanity.com/paper/2506.07227/) (2025)
- [Consistent Image Layout Editing with Diffusion Models](https://mlsanity.com/paper/2503.06419/) (2025)
- [VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition](https://mlsanity.com/paper/2412.06235/) (2024)
- [TaxaDiffusion: Progressively Trained Diffusion Model for Fine-Grained Species Generation](https://mlsanity.com/paper/2506.01923/) (2025)
- [On enhancing the robustness of Vision Transformers: Defensive Diffusion](https://mlsanity.com/paper/2305.08031/) (2023)
- [TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life](https://mlsanity.com/paper/2603.26128/) (2026)
- [Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion](https://mlsanity.com/paper/2507.23483/) (2025)
- [FluxFlow: Conservative Flow-Matching for Astronomical Image Super-Resolution](https://mlsanity.com/paper/2605.03749/) (2026)
- [Fine-grained Appearance Transfer with Diffusion Models](https://mlsanity.com/paper/2311.16513/) (2023)
- [Bridging the Gap Between Multimodal Foundation Models and World Models](https://mlsanity.com/paper/2510.03727/) (2025)
- [In-context Region-based Drag: Drag Any Region to Any Shape](https://mlsanity.com/paper/2606.25907/) (2026)

## Abstract (arXiv)

Our goal is to develop fine-grained real-image editing methods suitable for real-world applications. In this paper, we first summarize four requirements for these methods and propose a novel diffusion-based image editing framework with pixel-wise guidance that satisfies these requirements. Specifically, we train pixel-classifiers with a few annotated data and then infer the segmentation map of a target image. Users then manipulate the map to instruct how the image will be edited. We utilize a pre-trained diffusion model to generate edited images aligned with the user's intention with pixel-wise guidance. The effective combination of proposed guidance and other techniques enables highly controllable editing with preserving the outside of the edited area, which results in meeting our requirements. The experimental results demonstrate that our proposal outperforms the GAN-based method for editing quality and speed.