# Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Model

arXiv: https://arxiv.org/abs/2609.00862 · HTML: https://mlsanity.com/paper/2609.00862/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 30 of its 34 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [Path Integral Sampler: a stochastic control approach for sampling](https://mlsanity.com/paper/2111.15141/) (2021)
- [End-to-End Diffusion Latent Optimization Improves Classifier Guidance](https://mlsanity.com/paper/2303.13703/) (2023)
- [Test-time Alignment of Diffusion Models without Reward Over-optimization](https://mlsanity.com/paper/2501.05803/) (2025)
- [Variational Control for Guidance in Diffusion Models](https://mlsanity.com/paper/2502.03686/) (2025)
- [Gradpaint: Gradient-Guided Inpainting with Diffusion Models](https://mlsanity.com/paper/2309.09614/) (2023)
- [PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference](https://mlsanity.com/paper/2410.21966/) (2024)
- [LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling](https://mlsanity.com/paper/2502.03491/) (2025)
- [PixelHacker: Image Inpainting with Structural and Semantic Consistency](https://mlsanity.com/paper/2504.20438/) (2025)
- [Coherent and Multi-modality Image Inpainting via Latent Space Optimization](https://mlsanity.com/paper/2407.08019/) (2024)
- [Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance](https://mlsanity.com/paper/2512.18365/) (2025)
- [Calibrated Test-Time Guidance for Bayesian Inference](https://mlsanity.com/paper/2602.22428/) (2026)
- [FreeCond: Free Lunch in the Input Conditions of Text-Guided Inpainting](https://mlsanity.com/paper/2412.00427/) (2024)
- [HiGS: History-Guided Sampling for Plug-and-Play Enhancement of Diffusion Models](https://mlsanity.com/paper/2509.22300/) (2025)
- [InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting](https://mlsanity.com/paper/2603.23463/) (2026)
- [HarmonPaint: Harmonized Training-Free Diffusion Inpainting](https://mlsanity.com/paper/2507.16732/) (2025)

## Abstract (arXiv)

We show that a frozen generic text-to-image diffusion model can perform conditional inpainting across three evaluated natural-image domains with one fixed controller configuration, without inpainting-specific weight training, dataset-specific weight adaptation, or learned inpainting-specific conditioning channels. Step-PI augments known-region projection with boundary-interior latent feedback, persistent PI state, and a predefined four-field release schedule that modulates controller signals along the reverse trajectory. Developed only on Main35-disjoint CelebA-HQ pilots, the controller transfers unchanged to AFHQ and Places2. Across two field-identical comparisons on the same 3,500 cases, adding persistent state and replacing uniform release with the predefined schedule each improve all 15 dataset-metric cells; 95% bootstrap intervals exclude zero for all five metrics in both comparisons. In descriptive native-route comparisons, Step-PI leads LanPaint and PILOT (the closest evaluated training-free baselines using vanilla SD1.5) on all five equal-dataset macro metrics. Inpainting-trained systems retain the absolute metric leads but rely on substantial inpainting-specific offline optimization. Our method provides a complementary approach for repurposing a frozen generic text-to-image model for cross-domain inpainting through test-time latent control.