# In-Loop Model Adaptation with Coupled Latent-Noise Guidance for High-Fidelity Subject-Driven Text-to-Image Generation arXiv: https://arxiv.org/abs/2608.09244 ยท HTML: https://mlsanity.com/paper/2608.09244/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 28 of its 33 references that are in the map. - First posted on arXiv 10 Aug 2026. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 28 of 28) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022) - [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023) - [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022) - [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022) - [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023) - [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023) - [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023) - [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023) - [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022) - [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023) - [SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation](https://mlsanity.com/paper/2312.16272/) (2023) - [Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation](https://mlsanity.com/paper/2212.04145/) (2022) - [Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models](https://mlsanity.com/paper/2307.06925/) (2023) - [A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis](https://mlsanity.com/paper/2306.14544/) (2023) - [Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation](https://mlsanity.com/paper/2303.00914/) (2023) - [Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning](https://mlsanity.com/paper/2407.12164/) (2024) - [Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation](https://mlsanity.com/paper/2406.19341/) (2024) - [HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generation](https://mlsanity.com/paper/2410.08192/) (2024) - [Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation](https://mlsanity.com/paper/2408.13983/) (2024) - [Domain-Conditioned Transformer for Fully Test-time Adaptation](https://mlsanity.com/paper/2410.10442/) (2024) ## Abstract (arXiv) Text-to-image diffusion models have achieved remarkable success in generating high-quality images from a given text prompt. Subject-driven generation aims to synthesize customized images to mimic the appearance of subjects in given reference images within different visual contexts specified by the text prompts. The central challenge here is that, when the reference image changes, the diffusion model cannot efficiently adapt to different visual contexts while consistently maintaining the subject identity. Existing methods either train the model with a large domain-specific dataset or fine-tune the model using the reference image for hundreds of iterations before actual image generation. In this work, we explore a new approach, called textitIn-Loop Model Adaptation (IMA), which adapts the core diffusion model at each generation step during the actual process of image generation, without being trained on the reference image before the generation process. To this end, we establish a DDIM inversion chain that maps the reference image to a sequence of latent, as well as a text-to-image generation chain which generates the image from the text prompt only. We then introduce a masked latent consistency loss and a noise regularization loss to characterize the latent-noise difference between the diffusion model and these two chains at each generation step. This coupled latent-noise loss is used to guide the in-loop model adaptation to preserve the subject identity specified by the reference image while maintaining accurate alignment with the text prompt, resulting in high-fidelity text-to-image generation. Our extensive experiments demonstrate that our proposed IMA method significantly improves the performance of subject-driven text-to-image generation.