Disentangling Instruction Influence in Diffusion Transformers for Parallel Multi-Instruction-Guided Image Editing
On the map: Diffusion models → Image generation, text-to-image and editing
Abstract (arXiv)
Instruction-guided image editing enables users to specify modifications using natural language, offering more flexibility and control. Among existing frameworks, Diffusion Transformers (DiTs) outperform U-Net-based diffusion models in scalability and performance. However, while real-world scenarios often require concurrent execution of multiple instructions, step-by-step editing suffers from accumulated errors and degraded quality, and integrating multiple instructions with a single prompt usually results in incomplete edits due to instruction conflicts. We propose Instruction Influence Disentanglement (IID), a novel framework enabling parallel execution of multiple instructions in a single denoising process, designed for DiT-based models. By analyzing self-attention mechanisms in DiTs, we identify distinctive attention patterns in multi-instruction settings and derive instruction-specific attention masks to disentangle each instruction's influence. These masks guide the editing process to ensure localized modifications while preserving consistency in non-edited regions. Extensive experiments on open-source and custom datasets demonstrate that IID reduces diffusion steps while improving fidelity and instruction completion compared to existing baselines. The codes will be publicly released upon the acceptance of the paper.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 38 of its 43 references that are in the map. First posted on arXiv 7 Apr 2025.
Builds on
38 of 43 references.
Closest ancestors · 15
Same direction, newest first
- 2024Stable Flow: Vital Layers for Training-Free Image Editing21 Nov
- 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision11 Nov
- 2024SeedEdit: Align Image Re-Generation to Image Editing11 Nov
- 2024Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing24 Oct
- 2024OmniGen: Unified Image Generation17 Sep
- 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models11 Dec
- 2023Inversion-Free Image Editing with Natural Language7 Dec
- 2023Iterative Multi-granular Image Editing using Diffusion Models1 Sep
- 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
- 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
- 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
3 more
Other references · 23
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
11 more
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
- 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
- 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
- 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 214
- 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
- 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
- 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
- 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
- 2024Multi-Reward as Condition for Instruction-based Image Editingcited by 20
- 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer
- 2025CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editing
- 2025Image Editing As Programs with Diffusion Models
- 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback
- 2026Editing on the Generative Manifold: A Theoretical and Empirical Study of General Diffusion-Based Image Editing Trade-offs
- 2026Dual-Channel Attention Guidance for Training-Free Image Editing Control in Diffusion Transformers
- 2026MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance
- 2025Self-Attention Decomposition For Training Free Diffusion Editing