# Steering Rectified Flow Models in the Vector Field for Controlled Image Generation

arXiv: https://arxiv.org/abs/2412.00100 · HTML: https://mlsanity.com/paper/2412.00100/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Flow matching, time series and science, one of 14 directions in Diffusion models.
- Builds on 54 of its 58 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Placed in

- Diffusion models / [Flow matching, time series and science](https://mlsanity.com/diffusion/flow-matching-time-series-and-science/)

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model](https://mlsanity.com/paper/2303.09833/) (2023)
- [An Edit Friendly DDPM Noise Space: Inversion and Manipulations](https://mlsanity.com/paper/2304.06140/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency](https://mlsanity.com/paper/2307.08123/) (2023)

## Built on by (most cited first, 30 of 39)

- [FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems](https://mlsanity.com/paper/2503.08136/) (2025)
- [UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Models](https://mlsanity.com/paper/2504.13109/) (2025)
- [Solving Inverse Problems with FLAIR](https://mlsanity.com/paper/2506.02680/) (2025)
- [Diffusion models for inverse problems](https://mlsanity.com/paper/2508.01975/) (2025)
- [RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions](https://mlsanity.com/paper/2506.03448/) (2025)
- [EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignment](https://mlsanity.com/paper/2511.00804/) (2025)
- [InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem](https://mlsanity.com/paper/2512.05672/) (2025)
- [FlowLPS: Langevin-Proximal Sampling for Flow-based Inverse Problem Solvers](https://mlsanity.com/paper/2512.07150/) (2025)
- [OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policies](https://mlsanity.com/paper/2603.10052/) (2026)
- [Optimal Transport for Rectified Flow Image Editing: Unifying Inversion-Based and Direct Methods](https://mlsanity.com/paper/2508.02363/) (2025)
- [Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance](https://mlsanity.com/paper/2512.18365/) (2025)
- [Delta Rectified Flow Sampling for Text-to-Image Editing](https://mlsanity.com/paper/2509.05342/) (2025)
- [Training-Free Reward-Guided Image Editing via Trajectory Optimal Control](https://mlsanity.com/paper/2509.25845/) (2025)
- [SONIC: Spectral Optimization of Noise for Inpainting with Consistency](https://mlsanity.com/paper/2511.19985/) (2025)
- [FMPlug: Plug-In Foundation Flow-Matching Priors for Inverse Problems](https://mlsanity.com/paper/2508.00721/) (2025)
- [The Aging Multiverse: Generating Condition-Aware Facial Aging Tree via Training-Free Diffusion](https://mlsanity.com/paper/2506.21008/) (2025)
- [3D-Consistent Multi-View Editing by Correspondence Guidance](https://mlsanity.com/paper/2511.22228/) (2025)
- [Free Lunch for Stabilizing Rectified Flow Inversion](https://mlsanity.com/paper/2602.11850/) (2026)
- [FlowSteer: Conditioning Flow Field for Consistent Image Restoration](https://mlsanity.com/paper/2512.08125/) (2025)
- [Measurement-Consistent Langevin Corrector for Stabilizing Latent Diffusion Inverse Problem Solvers](https://mlsanity.com/paper/2601.04791/) (2026)
- [Learning Sewing Patterns via Latent Flow Matching of Implicit Fields](https://mlsanity.com/paper/2601.17740/) (2026)
- [Solving Inverse Problems with Flow-based Models via Model Predictive Control](https://mlsanity.com/paper/2601.23231/) (2026)
- [Tunable Latent Generative Priors for Compressed Sensing and Inverse Problems](https://mlsanity.com/paper/2603.07357/) (2026)
- [Flow-Corrected Shape Optimization: Taming Manifold Drift in High-Dimensional 3D Models](https://mlsanity.com/paper/2608.07199/) (2026)
- [Dual Caption Preference Optimization for Diffusion Models](https://mlsanity.com/paper/2502.06023/) (2025)
- [Guiding Diffusion with Deep Geometric Moments: Balancing Fidelity and Variation](https://mlsanity.com/paper/2505.12486/) (2025)
- [Saving Foundation Flow-Matching Priors for Inverse Problems](https://mlsanity.com/paper/2511.16520/) (2025)
- [Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment](https://mlsanity.com/paper/2511.16870/) (2025)
- [When Test-Time Guidance Is Enough: Fast Image and Video Editing with Diffusion Guidance](https://mlsanity.com/paper/2602.14157/) (2026)
- [RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection](https://mlsanity.com/paper/2602.19974/) (2026)

## Abstract (arXiv)

Diffusion models (DMs) excel in photorealism, image editing, and solving inverse problems, aided by classifier-free guidance and image inversion techniques. However, rectified flow models (RFMs) remain underexplored for these tasks. Existing DM-based methods often require additional training, lack generalization to pretrained latent models, underperform, and demand significant computational resources due to extensive backpropagation through ODE solvers and inversion processes. In this work, we first develop a theoretical and empirical understanding of the vector field dynamics of RFMs in efficiently guiding the denoising trajectory. Our findings reveal that we can navigate the vector field in a deterministic and gradient-free manner. Utilizing this property, we propose FlowChef, which leverages the vector field to steer the denoising trajectory for controlled image generation tasks, facilitated by gradient skipping. FlowChef is a unified framework for controlled image generation that, for the first time, simultaneously addresses classifier guidance, linear inverse problems, and image editing without the need for extra training, inversion, or intensive backpropagation. Finally, we perform extensive evaluations and show that FlowChef significantly outperforms baselines in terms of performance, memory, and time requirements, achieving new state-of-the-art results. Project Page: urlhttps://flowchef.github.io.