# Regularization by Texts for Latent Diffusion Inverse Solvers

arXiv: https://arxiv.org/abs/2311.15658 · HTML: https://mlsanity.com/paper/2311.15658/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [A Variational Perspective on Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2305.04391/) (2023)
- [Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency](https://mlsanity.com/paper/2307.08123/) (2023)
- [Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models](https://mlsanity.com/paper/2307.00619/) (2023)
- [Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2211.10655/) (2022)
- [Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems](https://mlsanity.com/paper/2303.05754/) (2023)
- [Delta Denoising Score](https://mlsanity.com/paper/2304.07090/) (2023)
- [Parallel Diffusion Models of Operator and Image for Blind Inverse Problems](https://mlsanity.com/paper/2211.10656/) (2022)
- [Controlling Vision-Language Models for Multi-Task Image Restoration](https://mlsanity.com/paper/2310.01018/) (2023)
- [Prompt-tuning latent diffusion models for inverse problems](https://mlsanity.com/paper/2310.01110/) (2023)
- [Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models](https://mlsanity.com/paper/2303.08440/) (2023)

## Built on by (most cited first, 30 of 32)

- [A Survey on Diffusion Models for Inverse Problems](https://mlsanity.com/paper/2410.00083/) (2024)
- [CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models](https://mlsanity.com/paper/2406.08070/) (2024)
- [FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems](https://mlsanity.com/paper/2503.08136/) (2025)
- [DreamSampler: Unifying Diffusion Sampling and Score Distillation for Image Manipulation](https://mlsanity.com/paper/2403.11415/) (2024)
- [SILO: Solving Inverse Problems with Latent Operators](https://mlsanity.com/paper/2501.11746/) (2025)
- [Minority-Focused Text-to-Image Generation via Prompt Optimization](https://mlsanity.com/paper/2410.07838/) (2024)
- [LATINO-PRO: LAtent consisTency INverse sOlver with PRompt Optimization](https://mlsanity.com/paper/2503.12615/) (2025)
- [Diffusion models for inverse problems](https://mlsanity.com/paper/2508.01975/) (2025)
- [VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models](https://mlsanity.com/paper/2412.00156/) (2024)
- [Invertible Diffusion Models for Compressed Sensing](https://mlsanity.com/paper/2403.17006/) (2024)
- [OmniSSR: Zero-shot Omnidirectional Image Super-Resolution using Stable Diffusion Model](https://mlsanity.com/paper/2404.10312/) (2024)
- [Blind Inversion using Latent Diffusion Priors](https://mlsanity.com/paper/2407.01027/) (2024)
- [Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment](https://mlsanity.com/paper/2505.18600/) (2025)
- [InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem](https://mlsanity.com/paper/2512.05672/) (2025)
- [MRPD: Undersampled MRI reconstruction by prompting a large latent diffusion model](https://mlsanity.com/paper/2402.10609/) (2024)
- [Noise is All You Need: Solving Linear Inverse Problems by Noise Combination Sampling with Diffusion Models](https://mlsanity.com/paper/2510.23633/) (2025)
- [Tell Me What You See: Text-Guided Real-World Image Denoising](https://mlsanity.com/paper/2312.10191/) (2023)
- [Ground-A-Score: Scaling Up the Score Distillation for Multi-Attribute Editing](https://mlsanity.com/paper/2403.13551/) (2024)
- [VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference](https://mlsanity.com/paper/2411.18929/) (2024)
- [Image Restoration via Diffusion Models with Dynamic Resolution](https://mlsanity.com/paper/2605.14267/) (2026)
- [DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model](https://mlsanity.com/paper/2408.07541/) (2024)
- [Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction](https://mlsanity.com/paper/2510.03352/) (2025)
- [Measurement-Consistent Langevin Corrector for Stabilizing Latent Diffusion Inverse Problem Solvers](https://mlsanity.com/paper/2601.04791/) (2026)
- [Language-Free Generative Editing from One Visual Example](https://mlsanity.com/paper/2603.25441/) (2026)
- [ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts](https://mlsanity.com/paper/2411.17077/) (2024)
- [LatentINDIGO: An INN-Guided Latent Diffusion Algorithm for Image Restoration](https://mlsanity.com/paper/2505.12935/) (2025)
- [ART-VITON: Measurement-Guided Latent Diffusion for Artifact-Free Virtual Try-On](https://mlsanity.com/paper/2509.25749/) (2025)
- [Active Learning for Conditional Generative Compressed Sensing](https://mlsanity.com/paper/2605.05435/) (2026)
- [Triadic Dynamics Aware Diffusion Posterior Sampling for Inverse Problems: Optimizing Guidance and Stochasticity Schedules](https://mlsanity.com/paper/2605.26470/) (2026)
- [Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References](https://mlsanity.com/paper/2608.26476/) (2026)

## Abstract (arXiv)

The recent development of diffusion models has led to significant progress in solving inverse problems by leveraging these models as powerful generative priors. However, challenges persist due to the ill-posed nature of such problems, often arising from ambiguities in measurements or intrinsic system symmetries. To address this, here we introduce a novel latent diffusion inverse solver, regularization by text (TReg), inspired by the human ability to resolve visual ambiguities through perceptual biases. TReg integrates textual descriptions of preconceptions about the solution during reverse diffusion sampling, dynamically reinforcing these descriptions through null-text optimization, which we refer to as adaptive negation. Our comprehensive experimental results demonstrate that TReg effectively mitigates ambiguity in inverse problems, improving both accuracy and efficiency.