# Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing arXiv: https://arxiv.org/abs/2508.07519 ยท HTML: https://mlsanity.com/paper/2508.07519/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 60 of its 70 references that are in the map. - First posted on arXiv 11 Aug 2025. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 60) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023) - [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021) - [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022) - [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021) - [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023) - [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022) ## Built on by (most cited first, 17 of 17) - [Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers](https://mlsanity.com/paper/2607.19139/) (2026) - [Plan-X: Instruct Video Generation via Semantic Planning](https://mlsanity.com/paper/2511.17986/) (2025) - [TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers](https://mlsanity.com/paper/2601.02211/) (2026) - [NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces](https://mlsanity.com/paper/2603.00180/) (2026) - [Large-scale Codec Avatars: The Unreasonable Effectiveness of Large-scale Avatar Pretraining](https://mlsanity.com/paper/2604.02320/) (2026) - [Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers](https://mlsanity.com/paper/2605.13974/) (2026) - [Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers](https://mlsanity.com/paper/2605.14270/) (2026) - [Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers for Text-to-Image Generation](https://mlsanity.com/paper/2602.06886/) (2026) - [ASemConsist: Adaptive Semantic Feature Control for Training-Free Identity-Consistent Generation](https://mlsanity.com/paper/2512.23245/) (2025) - [CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformers](https://mlsanity.com/paper/2512.24195/) (2025) - [Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers](https://mlsanity.com/paper/2603.13386/) (2026) - [Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation](https://mlsanity.com/paper/2605.29390/) (2026) - [Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation](https://mlsanity.com/paper/2606.06813/) (2026) - [BindEdit: Taming Attention Leakage for Precise Multi-Object Image Editing](https://mlsanity.com/paper/2606.18906/) (2026) - [MedDiT4SR: Tri-Stream Joint Adaptation of Pre-Trained Diffusion Transformers for Medical Image Super-Resolution](https://mlsanity.com/paper/2607.20598/) (2026) - [JewelTry: Mask-Free Scale Aware Jewelry Virtual Try-On](https://mlsanity.com/paper/2609.16626/) (2026) - [Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback](https://mlsanity.com/paper/2610.13941/) (2026) ## Abstract (arXiv) Transformer-based diffusion models have recently superseded traditional U-Net architectures, with multimodal diffusion transformers (MM-DiT) emerging as the dominant approach in state-of-the-art models like Stable Diffusion 3 and Flux.1. Previous approaches have relied on unidirectional cross-attention mechanisms, with information flowing from text embeddings to image latents. In contrast, MMDiT introduces a unified attention mechanism that concatenates input projections from both modalities and performs a single full attention operation, allowing bidirectional information flow between text and image branches. This architectural shift presents significant challenges for existing editing techniques. In this paper, we systematically analyze MM-DiT's attention mechanism by decomposing attention matrices into four distinct blocks, revealing their inherent characteristics. Through these analyses, we propose a robust, prompt-based image editing method for MM-DiT that supports global to local edits across various MM-DiT variants, including few-step models. We believe our findings bridge the gap between existing U-Net-based methods and emerging architectures, offering deeper insights into MMDiT's behavioral patterns.