# Unified Diffusion-Based Rigid and Non-Rigid Editing with Text and Image Guidance

arXiv: https://arxiv.org/abs/2401.02126 · HTML: https://mlsanity.com/paper/2401.02126/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 19 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Builds on (most cited first, 18 of 18)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models](https://mlsanity.com/paper/2305.16807/) (2023)
- [Diffusion-based Image Translation using Disentangled Style and Content Representation](https://mlsanity.com/paper/2209.15264/) (2022)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [Cross-Image Attention for Zero-Shot Appearance Transfer](https://mlsanity.com/paper/2311.03335/) (2023)
- [Improving Tuning-Free Real Image Editing with Proximal Guidance](https://mlsanity.com/paper/2306.05414/) (2023)
- [Z^*: Zero-shot Style Transfer via Attention Rearrangement](https://mlsanity.com/paper/2311.16491/) (2023)

## Built on by (most cited first, 7 of 7)

- [TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models](https://mlsanity.com/paper/2408.00735/) (2024)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [Closed-Form Concept Erasure via Double Projections](https://mlsanity.com/paper/2604.10032/) (2026)
- [Beyond Inserting: Learning Identity Embedding for Semantic-Fidelity Personalized Diffusion Generation](https://mlsanity.com/paper/2402.00631/) (2024)
- [Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing](https://mlsanity.com/paper/2402.08601/) (2024)
- [Semantix: An Energy Guided Sampler for Semantic Style Transfer](https://mlsanity.com/paper/2503.22344/) (2025)
- [Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking](https://mlsanity.com/paper/2608.14046/) (2026)

## Abstract (arXiv)

Existing text-to-image editing methods tend to excel either in rigid or non-rigid editing but encounter challenges when combining both, resulting in misaligned outputs with the provided text prompts. In addition, integrating reference images for control remains challenging. To address these issues, we present a versatile image editing framework capable of executing both rigid and non-rigid edits, guided by either textual prompts or reference images. We leverage a dual-path injection scheme to handle diverse editing scenarios and introduce an integrated self-attention mechanism for fusion of appearance and structural information. To mitigate potential visual artifacts, we further employ latent fusion techniques to adjust intermediate latents. Compared to previous work, our approach represents a significant advance in achieving precise and versatile image editing. Comprehensive experiments validate the efficacy of our method, showcasing competitive or superior results in text-based editing and appearance transfer tasks, encompassing both rigid and non-rigid settings.