# Learning Flow Fields in Attention for Controllable Person Image Generation

arXiv: https://arxiv.org/abs/2412.08486 · HTML: https://mlsanity.com/paper/2412.08486/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 78 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](https://mlsanity.com/paper/2305.10431/) (2023)
- [VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization](https://mlsanity.com/paper/2103.16874/) (2021)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [The Role of ImageNet Classes in Fr'echet Inception Distance](https://mlsanity.com/paper/2203.06026/) (2022)
- [StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On](https://mlsanity.com/paper/2312.01725/) (2023)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [Grounded Text-to-Image Synthesis with Attention Refocusing](https://mlsanity.com/paper/2306.05427/) (2023)
- [Dress Code: High-Resolution Multi-Category Virtual Try-On](https://mlsanity.com/paper/2204.08532/) (2022)
- [LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On](https://mlsanity.com/paper/2305.13501/) (2023)
- [OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on](https://mlsanity.com/paper/2403.01779/) (2024)
- [Controllable Person Image Synthesis with Attribute-Decomposed GAN](https://mlsanity.com/paper/2003.12267/) (2020)

## Built on by (most cited first, 27 of 27)

- [MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-on](https://mlsanity.com/paper/2505.21325/) (2025)
- [FastFit: Accelerating Multi-Reference Virtual Try-On via Cacheable Diffusion Models](https://mlsanity.com/paper/2508.20586/) (2025)
- [Voost: A Unified and Scalable Diffusion Transformer for Bidirectional Virtual Try-On and Try-Off](https://mlsanity.com/paper/2508.04825/) (2025)
- [Visual Persona: Foundation Model for Full-Body Human Customization](https://mlsanity.com/paper/2503.15406/) (2025)
- [Tstars-Tryon 1.0: Robust and Realistic Virtual Try-On for Diverse Fashion Items](https://mlsanity.com/paper/2604.19748/) (2026)
- [Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individuals](https://mlsanity.com/paper/2505.21062/) (2025)
- [DS-VTON: An Enhanced Dual-Scale Coarse-to-Fine Framework for Virtual Try-On](https://mlsanity.com/paper/2506.00908/) (2025)
- [FitControler: Toward Fit-Aware Virtual Try-On](https://mlsanity.com/paper/2512.24016/) (2025)
- [Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning](https://mlsanity.com/paper/2603.09512/) (2026)
- [OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework](https://mlsanity.com/paper/2603.19643/) (2026)
- [DirectTryOn: One-Step Virtual Try-On via Straightened Conditional Transport](https://mlsanity.com/paper/2605.12939/) (2026)
- [Oxygen-TryOn: Fashion-Native Foundation Model for Any-item Virtual Try-On](https://mlsanity.com/paper/2607.21694/) (2026)
- [Clothing agnostic Pre-inpainting Virtual Try-ON](https://mlsanity.com/paper/2509.17654/) (2025)
- [Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise](https://mlsanity.com/paper/2511.18775/) (2025)
- [Pro-Pose: Unpaired Full-Body Portrait Synthesis via Canonical UV Maps](https://mlsanity.com/paper/2512.17143/) (2025)
- [VTONGuard: Automatic Detection and Authentication of AI-Generated Virtual Try-On Content](https://mlsanity.com/paper/2601.13951/) (2026)
- [MV-Fashion: Towards Enabling Virtual Try-On and Size Estimation with Multi-View Paired Data](https://mlsanity.com/paper/2603.08147/) (2026)
- [Dress-ED: Instruction-Guided Editing for Virtual Try-On and Try-Off](https://mlsanity.com/paper/2603.22607/) (2026)
- [What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction](https://mlsanity.com/paper/2604.08716/) (2026)
- [Fashion130K: An E-commerce Fashion Dataset for Outfit Generation with Unified Multi-modal Condition](https://mlsanity.com/paper/2605.10127/) (2026)
- [MOFA-VTON: More Fashion Possibilities with Fine-Grained Adaptations in Virtual Try-On](https://mlsanity.com/paper/2606.11148/) (2026)
- [CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation](https://mlsanity.com/paper/2607.09362/) (2026)
- [Structure-Detail Decoupled Autoregressive Generation for Fast and High-Fidelity Virtual Try-On](https://mlsanity.com/paper/2607.11233/) (2026)
- [WearWow: Native 2K Multi-Garment Virtual Try-On via Adaptive Token Packing and Preference Alignment](https://mlsanity.com/paper/2607.19923/) (2026)
- [FlowVVTON: Flow-Guided Mask-Free Video Virtual Try-On](https://mlsanity.com/paper/2608.30450/) (2026)
- [TryOnReward: Learning Foveated Consistency for Reinforcement Fine-Tuning of Virtual Try-On](https://mlsanity.com/paper/2609.13259/) (2026)
- [TexTailor: Texture-Preserving Video Virtual Try-On via Adaptive Garment Conditioning](https://mlsanity.com/paper/2609.39335/) (2026)

## Abstract (arXiv)

Controllable person image generation aims to generate a person image conditioned on reference images, allowing precise control over the person's appearance or pose. However, prior methods often distort fine-grained textural details from the reference image, despite achieving high overall image quality. We attribute these distortions to inadequate attention to corresponding regions in the reference image. To address this, we thereby propose learning flow fields in attention (Leffa), which explicitly guides the target query to attend to the correct reference key in the attention layer during training. Specifically, it is realized via a regularization loss on top of the attention map within a diffusion-based baseline. Our extensive experiments show that Leffa achieves state-of-the-art performance in controlling appearance (virtual try-on) and pose (pose transfer), significantly reducing fine-grained detail distortion while maintaining high image quality. Additionally, we show that our loss is model-agnostic and can be used to improve the performance of other diffusion models.