mlsanityData to 7 Oct 2026

When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions

arXiv 2608.04820 ↗First posted 5 Aug 2026cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Face inpainting with diffusion models has recently achieved impressive visual quality, yet preserving identity fidelity under significant occlusion and conflicting text guidance remains a major challenge. To address this issue, we present Reference Semantic Inpainting for Face (ReSem-Face), a cascaded diffusion framework that introduces an explicit identity-conditioned semantic prior for multi-reference face inpainting. Our approach distills representative identity features from multiple references to reconstruct missing semantic regions, which then guide the diffusion process through a multi-stream conditioning architecture. This design provides strong semantic constraints when pixels are absent and stabilizes identity reconstruction while remaining compatible with prompt-driven edits. Experiments on CelebAHQ-IDI-5 and VGGFace2 demonstrate that ReSem-Face yields more reliable identity-preserving completion under severe semantic masks and improves text-controlled editing quality compared with representative baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 37 of its 51 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

37 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention6 Dec
  2. 2023PATMAT: Person Aware Tuning of Mask-Aware Transformer for Face Inpainting12 Apr
  3. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  8. 2018Demystifying MMD GANscited by 1,268
  9. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  10. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  11. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  12. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
19 more
  1. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  2. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  3. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  4. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  5. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  6. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 168
  7. 2022Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encodingcited by 83
  8. 2017Generative Face Completioncited by 79
  9. 2021TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformationscited by 48
  10. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 47
  11. 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillationcited by 21
  12. 2024RAD: Region-Aware Diffusion Models for Image Inpaintingcited by 19
  13. 2024ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restorationcited by 16
  14. 2025TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpaintingcited by 15
  15. 2024Improving Text-guided Object Inpainting with Semantic Pre-inpaintingcited by 12
  16. 2023Reference-Guided Large-Scale Face Inpainting with Identity and Texture Controlcited by 10
  17. 2019FCSR-GAN: Joint Face Completion and Super-resolution via Multi-task Learningcited by 7
  18. 2023TransRef: Multi-Scale Reference Embedding Transformer for Reference-Guided Image Inpaintingcited by 5
  19. 2026HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Imagescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Face inpainting with Identity Preserving Latent Diffusion Models
  2. 2023When StyleGAN Meets Stable Diffusion: a mathscrW_+ Adapter for Personalized Image Generation
  3. 2025Reference-Guided Identity Preserving Face Restoration
  4. 2025Robust ID-Specific Face Restoration via Alignment Learning
  5. 2026MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer
  6. 2025InstaFace: Identity-Preserving Facial Editing with Single Image Inference
  7. 2025High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning
  8. 2026Diff-ID: Identity Consistent Facial Image Generation and Morphing via Diffusion Models