# ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration

arXiv: https://arxiv.org/abs/2412.05043 · HTML: https://mlsanity.com/paper/2412.05043/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 4 of its 32 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 4 of 4)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)

## Built on by (most cited first, 16 of 16)

- [The Second Challenge on Real-World Face Restoration at NTIRE 2026: Methods and Results](https://mlsanity.com/paper/2604.10532/) (2026)
- [LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal](https://mlsanity.com/paper/2510.15868/) (2025)
- [Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion](https://mlsanity.com/paper/2512.23709/) (2025)
- [Kernel Density Steering: Inference-Time Scaling via Mode Seeking for Image Restoration](https://mlsanity.com/paper/2507.05604/) (2025)
- [Reference-Guided Identity Preserving Face Restoration](https://mlsanity.com/paper/2505.21905/) (2025)
- [HonestFace: Towards Honest Face Restoration with One-Step Diffusion Model](https://mlsanity.com/paper/2505.18469/) (2025)
- [RefSTAR: Blind Facial Image Restoration with Reference Selection, Transfer, and Reconstruction](https://mlsanity.com/paper/2507.10470/) (2025)
- [Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior](https://mlsanity.com/paper/2512.11121/) (2025)
- [Face2Scene: Using Facial Degradation as an Oracle for Diffusion-Based Scene Restoration](https://mlsanity.com/paper/2603.16570/) (2026)
- [MeInTime: Bridging Age Gap in Identity-Preserving Face Restoration](https://mlsanity.com/paper/2603.18645/) (2026)
- [TimeWeaver: Age-Consistent Reference-Based Face Restoration with Identity Preservation](https://mlsanity.com/paper/2603.22701/) (2026)
- [IConFace: Fine-Grained Identity Conditioning for Reference-Aware Face Restoration](https://mlsanity.com/paper/2605.02814/) (2026)
- [Reflection Separation from a Single Image via Joint Latent Diffusion](https://mlsanity.com/paper/2606.04107/) (2026)
- [RGFVR: Reference-Guided Face Video Restoration with Flow Matching](https://mlsanity.com/paper/2606.16401/) (2026)
- [When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions](https://mlsanity.com/paper/2608.04820/) (2026)
- [Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation](https://mlsanity.com/paper/2609.38019/) (2026)

## Abstract (arXiv)

While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may not accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs could be a promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM), we propose ReF-LDM, an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Additionally, we design a timestep-scaled identity loss, enabling our LDM-based model to focus on learning the discriminating features of human faces. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for reference-based face restoration models.