# PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor

arXiv: https://arxiv.org/abs/2303.17546 · HTML: https://mlsanity.com/paper/2303.17546/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 53 of its 63 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 53)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)

## Built on by (most cited first, 17 of 17)

- [OpenBias: Open-set Bias Detection in Text-to-Image Generative Models](https://mlsanity.com/paper/2404.07990/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regions](https://mlsanity.com/paper/2505.19352/) (2025)
- [LOTS of Fashion! Multi-Conditioning for Image Generation via Sketch-Text Pairing](https://mlsanity.com/paper/2507.22627/) (2025)
- [IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance](https://mlsanity.com/paper/2509.26231/) (2025)
- [GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models](https://mlsanity.com/paper/2408.16700/) (2024)
- [AGSwap: Overcoming Category Boundaries in Object Fusion via Adaptive Group Swapping](https://mlsanity.com/paper/2509.18699/) (2025)
- [DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.15194/) (2023)
- [Diverse Semantic Image Editing with Style Codes](https://mlsanity.com/paper/2309.13975/) (2023)
- [Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.04215/) (2025)
- [Shifting the Breaking Point of Flow Matching for Multi-Instance Editing](https://mlsanity.com/paper/2602.08749/) (2026)
- [FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models](https://mlsanity.com/paper/2404.13263/) (2024)
- [RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generation](https://mlsanity.com/paper/2507.02792/) (2025)
- [Region in Context: Text-condition Image editing with Human-like semantic reasoning](https://mlsanity.com/paper/2510.16772/) (2025)
- [Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation](https://mlsanity.com/paper/2602.18309/) (2026)
- [When Text-to-Image Helps Editing: The Effects of Conditioning During Denoising](https://mlsanity.com/paper/2610.01681/) (2026)

## Abstract (arXiv)

Generative image editing has recently witnessed extremely fast-paced growth. Some works use high-level conditioning such as text, while others use low-level conditioning. Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i.e. object-level image editing. In this work, we tackle the task by perceiving the images as an amalgamation of various objects and aim to control the properties of each object in a fine-grained manner. Out of these properties, we identify structure and appearance as the most intuitive to understand and useful for editing purposes. We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image. We show that having control over the properties of each object in an image leads to comprehensive editing capabilities. Our framework allows for various object-level editing operations on real images such as reference image-based appearance editing, free-form shape editing, adding objects, and variations. Thanks to our design, we do not require any inversion step. Additionally, we propose multimodal classifier-free guidance which enables editing images using both reference images and text when using our approach with foundational diffusion models. We validate the above claims by extensively evaluating our framework on both unconditional and foundational diffusion models. Please refer to https://vidit98.github.io/publication/conference-paper/pair_diff.html for code and model release.