mlsanityData to 7 Oct 2026

Amortized Guidance for Image Inpainting with Pretrained Diffusion Models

arXiv 2605.13010 ↗First posted 13 May 2026cs.CV cs.AI cs.SYcited by 0

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We study image inpainting with generative diffusion models. Existing methods typically either train dedicated task-specific models, or adapt a pretrained diffusion model separately for each masked image at deployment. We introduce a middle-ground model, termed Amortized Inpainting with Diffusion (AID), which keeps a pretrained diffusion backbone fixed, trains a small reusable guidance module offline, and then reuses it across masked images without per-instance optimization. We formulate it as a deterministic guidance problem with a supervised terminal objective. To make this problem learnable in high dimensions, we derive an auxiliary Gaussian formulation and prove that solving this randomized problem recovers the optimal deterministic guidance field. This bridge yields a principled continuous-time actor--critic algorithm for learning the guidance module in a fully data-driven manner. Empirically, on AFHQv2 and FFHQ under the pixel EDM pipeline and on ImageNet under the latent EDM2 pipeline, AID consistently improves the quality--speed trade-off over strong fixed-backbone and amortized inpainting baselines across multiple mask types, while adding less than one percent trainable overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 37 of its 43 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

37 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion11 Mar
  2. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
23 more
  1. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  2. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  3. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  4. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  5. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  7. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  8. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  9. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  10. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  11. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  12. 2024DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised h-transformcited by 44
  13. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  14. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  15. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  16. 2024Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problemscited by 13
  17. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 10
  18. 2024Regret of exploratory policy improvement and q-learningcited by 10
  19. 2024Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Studycited by 7
  20. 2024Solving Inverse Problems via Diffusion Optimal Controlcited by 7
  21. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 3
  22. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problemscited by 2
  23. 2025Continuous-Time Reinforcement Learning for Asset-Liability Managementcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Learning Sparse Masks for Diffusion-based Image Inpainting
  2. 2026Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Model
  3. 2026Learned End-to-End Guidance Schedules for Diffusion Models
  4. 2024RAD: Region-Aware Diffusion Models for Image Inpainting
  5. 2023Gradpaint: Gradient-Guided Inpainting with Diffusion Models
  6. 2024Diffusion-based image inpainting with internal learning
  7. 2025GuidPaint: Class-Guided Image Inpainting with Diffusion Models
  8. 20243D-Consistent Image Inpainting with Diffusion Models