mlsanityData to 7 Oct 2026

Prompt-tuning latent diffusion models for inverse problems

arXiv 2310.01110 ↗First posted 2 Oct 2023cs.LG cs.AI cs.CVcited by 70

On the map: Diffusion models → Inverse problems and image restoration

Abstract (arXiv)

We propose a new method for solving imaging inverse problems using text-to-image latent diffusion models as general priors. Existing methods using latent diffusion models for inverse problems typically rely on simple null text prompts, which can lead to suboptimal performance. To address this limitation, we introduce a method for prompt tuning, which jointly optimizes the text embedding on-the-fly while running the reverse diffusion process. This allows us to generate images that are more faithful to the diffusion prior. In addition, we propose a method to keep the evolution of latent variables within the range space of the encoder, by projection. This helps to reduce image artifacts, a major problem when using latent diffusion models instead of pixel-based diffusion models. Our combined method, called P2L, outperforms both image- and latent-diffusion model-based inverse problem solvers on a variety of tasks, such as super-resolution, deblurring, and inpainting.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models. Builds on 57 of its 75 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

57 of 75 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution1 Sep
  2. 2023Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency16 Jul
  3. 2023Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models2 Jul
  4. 2023Denoising Diffusion Models for Plug-and-Play Image Restoration15 May
  5. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolution11 May
  6. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Models7 May
  7. 2023Score-Based Diffusion Models as Principled Priors for Inverse Imaging23 Apr
  8. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration20 Mar
  9. 2023Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems10 Mar
  10. 2023Unlimited-Size Diffusion Restoration1 Mar
  11. 2023GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration30 Jan
  12. 2022Parallel Diffusion Models of Operator and Image for Blind Inverse Problems19 Nov
3 more
  1. 2022Diffusion Posterior Sampling for General Noisy Inverse Problems29 Sep
  2. 2022Denoising Diffusion Restoration Models27 Jan
  3. 2021Deblurring via Stochastic Refinement5 Dec

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
30 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  12. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  13. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  14. 2015Automatic differentiation in machine learning: a surveycited by 846
  15. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  16. 2017The Perception-Distortion Tradeoffcited by 582
  17. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  18. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  19. 2017Compressed Sensing using Generative Modelscited by 337
  20. 2021Score-based diffusion models for accelerated MRIcited by 308
  21. 2016The Little Engine that Could: Regularization by Denoising (RED)cited by 303
  22. 2020Deep Learning Techniques for Inverse Problems in Imagingcited by 290
  23. 2021SNIPS: Solving Noisy Inverse Problems Stochasticallycited by 191
  24. 2022Solving 3D Inverse Problems using Pre-trained 2D Diffusion Modelscited by 130
  25. 2022Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imagingcited by 94
  26. 2023Mixture of Diffusers for scene composition and high resolution image generationcited by 56
  27. 2023Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Modelscited by 53
  28. 2020Solving Inverse Problems with a Flow-based Noise Modelcited by 31
  29. 2021Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Priorcited by 24
  30. 2022Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse Problemscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 28; 2025: 22; 2026: 17.

70 papers.

Sort:

By area:

  1. 2024A Survey on Diffusion Models for Inverse Problemscited by 173
  2. 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Modelscited by 138
  3. 2023Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusioncited by 78
  4. 2025FlowDPS: Flow-Driven Posterior Sampling for Inverse Problemscited by 65
  5. 2024Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusioncited by 47
  6. 2024Denoising: A Powerful Building-Block for Imaging, Inverse Problems, and Machine Learningcited by 35
  7. 2023Regularization by Texts for Latent Diffusion Inverse Solverscited by 31
  8. 2024You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillationcited by 23
  9. 2025Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posteriorcited by 20
  10. 2024Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problemscited by 18
  11. 2024Minority-Focused Text-to-Image Generation via Prompt Optimizationcited by 18
  12. 2024Optical-Flow Guided Prompt Optimization for Coherent Video Generationcited by 17
38 more
  1. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  2. 2024Repulsive Latent Score Distillation for Solving Inverse Problemscited by 15
  3. 2024MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restorationcited by 15
  4. 2024ZoomLDM: Latent Diffusion Model for multi-scale image generationcited by 15
  5. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  6. 2025Diffusion models for inverse problemscited by 15
  7. 2024Diffusion Posterior Proximal Sampling for Image Restorationcited by 14
  8. 2025LATINO-PRO: LAtent consisTency INverse sOlver with PRompt Optimizationcited by 14
  9. 2024VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Modelscited by 13
  10. 2024Invertible Diffusion Models for Compressed Sensingcited by 12
  11. 2024OmniSSR: Zero-shot Omnidirectional Image Super-Resolution using Stable Diffusion Modelcited by 11
  12. 2024Blind Inversion using Latent Diffusion Priorscited by 10
  13. 2024Fast constrained sampling in pre-trained diffusion modelscited by 9
  14. 2024Consistency Model is an Effective Posterior Sample Approximation for Diffusion Inverse Solverscited by 8
  15. 2025Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignmentcited by 8
  16. 2025Whitened Score Diffusion: A Structured Prior for Imaging Inverse Problemscited by 6
  17. 2025Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Modelscited by 6
  18. 2025InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problemcited by 6
  19. 2026Robust Inference-Time Steering of Protein Diffusion Models via Embedding Optimizationcited by 6
  20. 2024Particle-Filtering-based Latent Diffusion for Inverse Problemscited by 5
  21. 2025Enhancing Diffusion Model Stability for Image Restoration via Gradient Managementcited by 5
  22. 2025Test-Time Anchoring for Discrete Diffusion Posterior Samplingcited by 5
  23. 2025Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Modelscited by 5
  24. 2024Efficient Diffusion-Driven Corruption Editor for Test-Time Adaptationcited by 4
  25. 2024Dreamguider: Improved Training free Diffusion-based Conditional Generationcited by 4
  26. 2024G2D2: Gradient-Guided Discrete Diffusion for Inverse Problem Solvingcited by 4
  27. 2025CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbationcited by 3
  28. 2023Tell Me What You See: Text-Guided Real-World Image Denoisingcited by 2
  29. 2024ODE-DPS: ODE-based Diffusion Posterior Sampling for Inverse Problems in Partial Differential Equationcited by 2
  30. 2024VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inferencecited by 2
  31. 2025Latent Refinement via Flow Matching for Training-free Linear Inverse Problem Solvingcited by 2
  32. 2026Entropy Aware Reward Guidance for Diffusion Language Model Alignmentcited by 2
  33. 2026Cross-Modal Guidance for Fast Diffusion-Based Computed Tomographycited by 2
  34. 2026Image Restoration via Diffusion Models with Dynamic Resolutioncited by 2
  35. 2024DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Modelcited by 1
  36. 2024Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusioncited by 1
  37. 2025Inference-Time Search Using Side Information for Diffusion-Based Image Reconstructioncited by 1
  38. 2026Measurement-Consistent Langevin Corrector for Stabilizing Latent Diffusion Inverse Problem Solverscited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
  2. 2025SILO: Solving Inverse Problems with Latent Operators
  3. 2026Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives
  4. 2024Enhancing and Accelerating Diffusion-Based Inverse Problem Solving through Measurements Optimization
  5. 2026Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation
  6. 2025Solving Inverse Problems with FLAIR
  7. 2026Your Pre-trained Diffusion Model Secretly Knows Restoration
  8. 2026Principled Design of Diffusion-based Optimizers for Inverse Problems