# Mind the RefGAP: Correcting Reference Attention in Diffusion-Based Visual Editing

arXiv: https://arxiv.org/abs/2609.35708 · HTML: https://mlsanity.com/paper/2609.35708/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 18 of 18)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization](https://mlsanity.com/paper/2103.16874/) (2021)
- [FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition](https://mlsanity.com/paper/2405.13870/) (2024)
- [Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator](https://mlsanity.com/paper/2411.15466/) (2024)
- [Group Relative Attention Guidance for Image Editing](https://mlsanity.com/paper/2510.24657/) (2025)
- [Dual-Channel Attention Guidance for Training-Free Image Editing Control in Diffusion Transformers](https://mlsanity.com/paper/2602.18022/) (2026)
- [LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing](https://mlsanity.com/paper/2606.06042/) (2026)
- [RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance](https://mlsanity.com/paper/2405.17661/) (2024)
- [JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion](https://mlsanity.com/paper/2608.03974/) (2026)
- [Head Similarity: Modeling Structured Whole-Head Appearance Beyond Face Recognition](https://mlsanity.com/paper/2605.07766/) (2026)

## Abstract (arXiv)

Reference-guided diffusion editors struggle to faithfully reproduce user-provided references. We identify a potential bottleneck in diffusion editors: many methods provide limited reference-attention allocation. For example, in LoomVideo, edit-region queries assign less than 1% of their attention mass to the reference. We introduce RefGAP, a training-free correction that determines logit-offset magnitudes online at each layer from the reference-attention mass measured during the forward pass. Positive offsets to reference logits strengthen reference usage by edit-region queries, while negative offsets for keep-region queries limit reference-induced changes outside the edit. Two global coefficients control the correction; they are selected once on validation data from four development diffusion editors and held fixed. Across seven diffusion-based image/video editors, RefGAP improves identity fidelity in head swapping and face swapping. RefGAP achieves a fidelity-preservation trade-off comparable to separately tuned constant edit-side biases, without per-approach strength sweeps. Additional experiments on virtual try-on and background replacement evaluate transfer beyond identity editing.