# RealFill: Reference-Driven Generation for Authentic Image Completion

arXiv: https://arxiv.org/abs/2309.16668 · HTML: https://mlsanity.com/paper/2309.16668/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 60 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)

## Built on by (most cited first, 30 of 59)

- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [Improving Diffusion Models for Authentic Virtual Try-on in the Wild](https://mlsanity.com/paper/2403.05139/) (2024)
- [UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics](https://mlsanity.com/paper/2412.07774/) (2024)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning](https://mlsanity.com/paper/2411.05003/) (2024)
- [Zero-shot Image Editing with Reference Imitation](https://mlsanity.com/paper/2406.07547/) (2024)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [NeRFiller: Completing Scenes via Generative 3D Inpainting](https://mlsanity.com/paper/2312.04560/) (2023)
- [Readout Guidance: Learning Control from Diffusion Features](https://mlsanity.com/paper/2312.02150/) (2023)
- [RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control](https://mlsanity.com/paper/2405.17401/) (2024)
- [Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing](https://mlsanity.com/paper/2503.19385/) (2025)
- [Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models](https://mlsanity.com/paper/2404.04243/) (2024)
- [Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering](https://mlsanity.com/paper/2408.09702/) (2024)
- [RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting](https://mlsanity.com/paper/2404.10765/) (2024)
- [AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360deg Unbounded Scene Inpainting](https://mlsanity.com/paper/2502.05176/) (2025)
- [CLiC: Concept Learning in Context](https://mlsanity.com/paper/2311.17083/) (2023)
- [Taming Latent Diffusion Model for Neural Radiance Field Inpainting](https://mlsanity.com/paper/2404.09995/) (2024)
- [Magic Insert: Style-Aware Drag-and-Drop](https://mlsanity.com/paper/2407.02489/) (2024)
- [MangaNinja: Line Art Colorization with Precise Reference Following](https://mlsanity.com/paper/2501.08332/) (2025)
- [RI3D: Few-Shot Gaussian Splatting With Repair and Inpainting Diffusion Priors](https://mlsanity.com/paper/2503.10860/) (2025)
- [Generate E-commerce Product Background by Integrating Category Commonality and Personalized Style](https://mlsanity.com/paper/2312.13309/) (2023)
- [PuzzleAvatar: Assembling 3D Avatars from Personal Albums](https://mlsanity.com/paper/2405.14869/) (2024)
- [Generative Powers of Ten](https://mlsanity.com/paper/2312.02149/) (2023)
- [PFStorer: Personalized Face Restoration and Super-Resolution](https://mlsanity.com/paper/2403.08436/) (2024)
- [GS-VTON: Controllable 3D Virtual Try-on with Gaussian Splatting](https://mlsanity.com/paper/2410.05259/) (2024)
- [DefectFill: Realistic Defect Generation with Inpainting Diffusion Model for Visual Inspection](https://mlsanity.com/paper/2503.13985/) (2025)
- [IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement](https://mlsanity.com/paper/2503.04501/) (2025)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collections](https://mlsanity.com/paper/2509.24817/) (2025)
- [LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal](https://mlsanity.com/paper/2510.15868/) (2025)

## Abstract (arXiv)

Recent advances in generative imagery have brought forth outpainting and inpainting models that can produce high-quality, plausible image content in unknown regions. However, the content these models hallucinate is necessarily inauthentic, since they are unaware of the true scene. In this work, we propose RealFill, a novel generative approach for image completion that fills in missing regions of an image with the content that should have been there. RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. These reference images do not have to be aligned with the target image, and can be taken with drastically varying viewpoints, lighting conditions, camera apertures, or image styles. Once personalized, RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene. We evaluate RealFill on a new image completion benchmark that covers a set of diverse and challenging scenarios, and find that it outperforms existing approaches by a large margin. Project page: https://realfill.github.io