# TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations

arXiv: https://arxiv.org/abs/2103.15982 · HTML: https://mlsanity.com/paper/2103.15982/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 78 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 30 of 43)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Deep Bilateral Learning for Real-Time Image Enhancement](https://mlsanity.com/paper/1707.02880/) (2017)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Deep Image Homography Estimation](https://mlsanity.com/paper/1606.03798/) (2016)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting](https://mlsanity.com/paper/2005.09704/) (2020)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)
- [Foreground-aware Image Inpainting](https://mlsanity.com/paper/1901.05945/) (2019)
- [High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling](https://mlsanity.com/paper/2005.11742/) (2020)

## Built on by (most cited first, 30 of 48)

- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics](https://mlsanity.com/paper/2412.07774/) (2024)
- [RealFill: Reference-Driven Generation for Authentic Image Completion](https://mlsanity.com/paper/2309.16668/) (2023)
- [Reference-guided Controllable Inpainting of Neural Radiance Fields](https://mlsanity.com/paper/2304.09677/) (2023)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing](https://mlsanity.com/paper/2408.08000/) (2024)
- [Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand](https://mlsanity.com/paper/2208.03382/) (2022)
- [LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model](https://mlsanity.com/paper/2305.11577/) (2023)
- [Few-Shot Head Swapping in the Wild](https://mlsanity.com/paper/2204.13100/) (2022)
- [Partially Does It: Towards Scene-Level FG-SBIR with Partial Input](https://mlsanity.com/paper/2203.14804/) (2022)
- [Reference-Guided Large-Scale Face Inpainting with Identity and Texture Control](https://mlsanity.com/paper/2303.07014/) (2023)
- [GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding](https://mlsanity.com/paper/2201.08131/) (2022)
- [Shape-guided Object Inpainting](https://mlsanity.com/paper/2204.07845/) (2022)
- [CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models](https://mlsanity.com/paper/2501.02355/) (2025)
- [ReGO: Reference-Guided Outpainting for Scenery Image](https://mlsanity.com/paper/2106.10601/) (2021)
- [Geometry-Aware Diffusion Models for Multiview Scene Inpainting](https://mlsanity.com/paper/2502.13335/) (2025)
- [Learning Second-Order Attentive Context for Efficient Correspondence Pruning](https://mlsanity.com/paper/2303.15761/) (2023)
- [TransRef: Multi-Scale Reference Embedding Transformer for Reference-Guided Image Inpainting](https://mlsanity.com/paper/2306.11528/) (2023)
- [Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment](https://mlsanity.com/paper/2412.00306/) (2024)
- [GeoComplete: Geometry-Aware Diffusion for Reference-Driven Image Completion](https://mlsanity.com/paper/2510.03110/) (2025)
- [Learning Continuous Implicit Representation for Near-Periodic Patterns](https://mlsanity.com/paper/2208.12278/) (2022)
- [Point-and-Shoot All-in-Focus Photo Synthesis from Smartphone Camera Pair](https://mlsanity.com/paper/2304.04917/) (2023)
- [PATMAT: Person Aware Tuning of Mask-Aware Transformer for Face Inpainting](https://mlsanity.com/paper/2304.06107/) (2023)
- [Modification Takes Courage: Seamless Image Stitching via Reference-Driven Inpainting](https://mlsanity.com/paper/2411.10309/) (2024)
- [3DFill:Reference-guided Image Inpainting by Self-supervised 3D Image Alignment](https://mlsanity.com/paper/2211.04831/) (2022)
- [Reference-based Painterly Inpainting via Diffusion: Crossing the Wild Reference Domain Gap](https://mlsanity.com/paper/2307.10584/) (2023)
- [SuperInpaint: Learning Detail-Enhanced Attentional Implicit Representation for Super-resolutional Image Inpainting](https://mlsanity.com/paper/2307.14489/) (2023)
- [DiGA3D: Coarse-to-Fine Diffusional Propagation of Geometry and Appearance for Versatile 3D Inpainting](https://mlsanity.com/paper/2507.00429/) (2025)
- [Multi-feature Co-learning for Image Inpainting](https://mlsanity.com/paper/2205.10578/) (2022)
- [Inharmonious Region Localization by Magnifying Domain Discrepancy](https://mlsanity.com/paper/2209.15368/) (2022)

## Abstract (arXiv)

Image inpainting is the task of plausibly restoring missing pixels within a hole region that is to be removed from a target image. Most existing technologies exploit patch similarities within the image, or leverage large-scale training data to fill the hole using learned semantic and texture information. However, due to the ill-posed nature of the inpainting task, such methods struggle to complete larger holes containing complicated scenes. In this paper, we propose TransFill, a multi-homography transformed fusion method to fill the hole by referring to another source image that shares scene contents with the target image. We first align the source image to the target image by estimating multiple homographies guided by different depth levels. We then learn to adjust the color and apply a pixel-level warping to each homography-warped source image to make it more consistent with the target. Finally, a pixel-level fusion module is learned to selectively merge the different proposals. Our method achieves state-of-the-art performance on pairs of images across a variety of wide baselines and color differences, and generalizes to user-provided image pairs.