# Latent Space Editing in Transformer-Based Flow Matching

arXiv: https://arxiv.org/abs/2312.10825 · HTML: https://mlsanity.com/paper/2312.10825/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 17 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [An Edit Friendly DDPM Noise Space: Inversion and Manipulations](https://mlsanity.com/paper/2304.06140/) (2023)
- [Scalable Adaptive Computation for Iterative Generation](https://mlsanity.com/paper/2212.11972/) (2022)
- [Minimizing Trajectory Curvature of ODE-based Generative Models](https://mlsanity.com/paper/2301.12003/) (2023)
- [DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2304.06648/) (2023)
- [Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models](https://mlsanity.com/paper/2303.11073/) (2023)
- [Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model](https://mlsanity.com/paper/2208.07791/) (2022)
- [Fast Point Cloud Generation with Straight Flows](https://mlsanity.com/paper/2212.01747/) (2022)
- [Self-Guided Diffusion Models](https://mlsanity.com/paper/2210.06462/) (2022)
- [Unsupervised Discovery of Semantic Latent Directions in Diffusion Models](https://mlsanity.com/paper/2302.12469/) (2023)

## Built on by (most cited first, 30 of 32)

- [ZigMa: A DiT-style Zigzag Mamba Diffusion Model](https://mlsanity.com/paper/2403.13802/) (2024)
- [DepthFM: Fast Monocular Depth Estimation with Flow Matching](https://mlsanity.com/paper/2403.13788/) (2024)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [Categorical Flow Matching on Statistical Manifolds](https://mlsanity.com/paper/2405.16441/) (2024)
- [Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions](https://mlsanity.com/paper/2403.17064/) (2024)
- [UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Models](https://mlsanity.com/paper/2504.13109/) (2025)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing](https://mlsanity.com/paper/2411.15843/) (2024)
- [TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training](https://mlsanity.com/paper/2501.04765/) (2025)
- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [Self-Corrected Flow Distillation for Consistent One-Step and Few-Step Text-to-Image Generation](https://mlsanity.com/paper/2412.16906/) (2024)
- [SplitFlow: Flow Decomposition for Inversion-Free Text-to-Image Editing](https://mlsanity.com/paper/2510.25970/) (2025)
- [MaskFlow: Discrete Flows For Flexible and Efficient Long Video Generation](https://mlsanity.com/paper/2502.11234/) (2025)
- [Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equation](https://mlsanity.com/paper/2402.18839/) (2024)
- [DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generation](https://mlsanity.com/paper/2411.04168/) (2024)
- [[MASK] is All You Need](https://mlsanity.com/paper/2412.06787/) (2024)
- [Designing a Conditional Prior Distribution for Flow-Based Generative Models](https://mlsanity.com/paper/2502.09611/) (2025)
- [Flow Matching is Adaptive to Manifold Structures](https://mlsanity.com/paper/2602.22486/) (2026)
- [From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech](https://mlsanity.com/paper/2503.16956/) (2025)
- [SCFlow: Implicitly Learning Style and Content Disentanglement with Flow Models](https://mlsanity.com/paper/2508.03402/) (2025)
- [Purrception: Variational Flow Matching for Vector-Quantized Image Generation](https://mlsanity.com/paper/2510.01478/) (2025)
- [Beyond Image Prior: Embedding Noise Prior into Conditional Denoising Transformer](https://mlsanity.com/paper/2407.09094/) (2024)
- [Diffusion Bridge or Flow Matching? A Unifying Framework and Comparative Analysis](https://mlsanity.com/paper/2509.24531/) (2025)
- [CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution](https://mlsanity.com/paper/2503.14272/) (2025)
- [3D-Consistent Multi-View Editing by Correspondence Guidance](https://mlsanity.com/paper/2511.22228/) (2025)
- [Measure-to-measure Regression with Transformers](https://mlsanity.com/paper/2605.28075/) (2026)
- [LGCC: Enhancing Flow Matching Based Text-Guided Image Editing with Local Gaussian Coupling and Context Consistency](https://mlsanity.com/paper/2511.01894/) (2025)
- [LensStyle: Learning the Optical Aesthetics for Controllable Stylized Lens Effect Rendering](https://mlsanity.com/paper/2609.04939/) (2026)
- [Latent Iterative Refinement Flow: A Geometric Constrained Approach for Few-Shot Generation](https://mlsanity.com/paper/2509.19903/) (2025)
- [Inference-time Trajectory Optimization for Structure-Preserving Manga Image Editing](https://mlsanity.com/paper/2603.27790/) (2026)

## Abstract (arXiv)

This paper strives for image editing via generative models. Flow Matching is an emerging generative modeling technique that offers the advantage of simple and efficient training. Simultaneously, a new transformer-based U-ViT has recently been proposed to replace the commonly used UNet for better scalability and performance in generative modeling. Hence, Flow Matching with a transformer backbone offers the potential for scalable and high-quality generative modeling, but their latent structure and editing ability are as of yet unknown. Hence, we adopt this setting and explore how to edit images through latent space manipulation. We introduce an editing space, which we call u-space, that can be manipulated in a controllable, accumulative, and composable manner. Additionally, we propose a tailored sampling solution to enable sampling with the more efficient adaptive step-size ODE solvers. Lastly, we put forth a straightforward yet powerful method for achieving fine-grained and nuanced editing using text prompts. Our framework is simple and efficient, all while being highly effective at editing images while preserving the essence of the original content. Our code will be publicly available at https://taohu.me/lfm/