# Differential Diffusion: Giving Each Pixel Its Strength

arXiv: https://arxiv.org/abs/2306.00950 · HTML: https://mlsanity.com/paper/2306.00950/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 52 of its 73 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)

## Built on by (most cited first, 29 of 29)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [PhysAnimator: Physics-Guided Generative Cartoon Animation](https://mlsanity.com/paper/2501.16550/) (2025)
- [3D Mesh Editing using Masked LRMs](https://mlsanity.com/paper/2412.08641/) (2024)
- [Tiled Diffusion](https://mlsanity.com/paper/2412.15185/) (2024)
- [Image Editing with Diffusion Models: A Survey](https://mlsanity.com/paper/2504.13226/) (2025)
- [SemanticDraw: Towards Real-Time Interactive Content Creation from Image Diffusion Models](https://mlsanity.com/paper/2403.09055/) (2024)
- [GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting](https://mlsanity.com/paper/2504.10001/) (2025)
- [ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detection](https://mlsanity.com/paper/2510.15783/) (2025)
- [Controllable Image Synthesis of Industrial Data Using Stable Diffusion](https://mlsanity.com/paper/2401.03152/) (2024)
- [PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery](https://mlsanity.com/paper/2501.09826/) (2025)
- [Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model](https://mlsanity.com/paper/2404.14951/) (2024)
- [Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models](https://mlsanity.com/paper/2512.05198/) (2025)
- [TAP: A Token-Adaptive Predictor Framework for Training-Free Diffusion Acceleration](https://mlsanity.com/paper/2603.03792/) (2026)
- [Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine](https://mlsanity.com/paper/2511.13713/) (2025)
- [ComboStoc: Combinatorial Stochasticity for Diffusion Generative Models](https://mlsanity.com/paper/2405.13729/) (2024)
- [RoCoTex: A Robust Method for Consistent Texture Synthesis with Diffusion Models](https://mlsanity.com/paper/2409.19989/) (2024)
- [SPICE: A Synergistic, Precise, Iterative, and Customizable Image Editing Workflow](https://mlsanity.com/paper/2504.09697/) (2025)
- [CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing](https://mlsanity.com/paper/2412.13565/) (2024)
- [IlluSign: Illustrating Sign Language Videos by Leveraging the Attention Mechanism](https://mlsanity.com/paper/2504.10822/) (2025)
- [SynSpill: Improved Industrial Spill Detection With Synthetic Data](https://mlsanity.com/paper/2508.10171/) (2025)
- [Colorful-Noise: Training-Free Low-Frequency Noise Manipulation for Color-Based Conditional Image Generation](https://mlsanity.com/paper/2605.00548/) (2026)
- [TD-Paint: Faster Diffusion Inpainting Through Time Aware Pixel Conditioning](https://mlsanity.com/paper/2410.09306/) (2024)
- [SceneTextStylizer: A Training-Free Scene Text Style Transfer Framework with Diffusion Model](https://mlsanity.com/paper/2510.10910/) (2025)
- [studentSplat: Your Student Model Learns Single-view 3D Gaussian Splatting](https://mlsanity.com/paper/2601.11772/) (2026)
- [CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation](https://mlsanity.com/paper/2607.09362/) (2026)
- [Text-based Tactile Graphics Generation for the Visually Impaired](https://mlsanity.com/paper/2607.22674/) (2026)
- [Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength](https://mlsanity.com/paper/2610.00359/) (2026)
- [NOFACE3D: NOise-scheduled Focused Anchoring and Controlled Enhancement for 3D Generation](https://mlsanity.com/paper/2610.13050/) (2026)
- [DepthTile: Depth-Guided Existing-Image Self-Tiling with Input-Side Tileability Assessment](https://mlsanity.com/paper/2610.16278/) (2026)

## Abstract (arXiv)

Diffusion models have revolutionized image generation and editing, producing state-of-the-art results in conditioned and unconditioned image synthesis. While current techniques enable user control over the degree of change in an image edit, the controllability is limited to global changes over an entire edited region. This paper introduces a novel framework that enables customization of the amount of change per pixel or per image region. Our framework can be integrated into any existing diffusion model, enhancing it with this capability. Such granular control on the quantity of change opens up a diverse array of new editing capabilities, such as control of the extent to which individual objects are modified, or the ability to introduce gradual spatial changes. Furthermore, we showcase the framework's effectiveness in soft-inpainting -- the completion of portions of an image while subtly adjusting the surrounding areas to ensure seamless integration. Additionally, we introduce a new tool for exploring the effects of different change quantities. Our framework operates solely during inference, requiring no model training or fine-tuning. We demonstrate our method with the current open state-of-the-art models, and validate it via both quantitative and qualitative comparisons, and a user study. Our code is available at: https://github.com/exx8/differential-diffusion