# Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

arXiv: https://arxiv.org/abs/2103.01648 · HTML: https://mlsanity.com/paper/2103.01648/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 81 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 30 of 51)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Compressed Sensing using Generative Models](https://mlsanity.com/paper/1703.03208/) (2017)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [Plug-and-Play Methods Provably Converge with Properly Trained Denoisers](https://mlsanity.com/paper/1905.05406/) (2019)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Flows for simultaneous manifold learning and density estimation](https://mlsanity.com/paper/2003.13913/) (2020)
- [Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems](https://mlsanity.com/paper/1704.03488/) (2017)
- [Regularization by Denoising: Clarifications and New Interpretations](https://mlsanity.com/paper/1806.02296/) (2018)
- [DeepISP: Towards Learning an End-to-End Image Processing Pipeline](https://mlsanity.com/paper/1801.06724/) (2018)
- [Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation](https://mlsanity.com/paper/1512.07331/) (2015)
- [CNN-Based Projected Gradient Descent for Consistent Image Reconstruction](https://mlsanity.com/paper/1709.01809/) (2017)
- [Learning Maximally Monotone Operators for Image Recovery](https://mlsanity.com/paper/2012.13247/) (2020)
- [Plug-and-Play Unplugged: Optimization Free Reconstruction using Consensus Equilibrium](https://mlsanity.com/paper/1705.08983/) (2017)

## Built on by (most cited first, 24 of 24)

- [Gradient Step Denoiser for convergent Plug-and-Play](https://mlsanity.com/paper/2110.03220/) (2021)
- [Proximal Denoiser for Convergent Plug-and-Play Optimization with Nonconvex Regularization](https://mlsanity.com/paper/2201.13256/) (2022)
- [Prompt-tuning latent diffusion models for inverse problems](https://mlsanity.com/paper/2310.01110/) (2023)
- [Learned reconstruction methods with convergence guarantees](https://mlsanity.com/paper/2206.05431/) (2022)
- [Generalized Normalizing Flows via Markov Chains](https://mlsanity.com/paper/2111.12506/) (2021)
- [Manifold Learning by Mixture Models of VAEs for Inverse Problems](https://mlsanity.com/paper/2303.15244/) (2023)
- [LATINO-PRO: LAtent consisTency INverse sOlver with PRompt Optimization](https://mlsanity.com/paper/2503.12615/) (2025)
- [Score Priors Guided Deep Variational Inference for Unsupervised Real-World Single Image Denoising](https://mlsanity.com/paper/2308.04682/) (2023)
- [Bayesian Inversion for Nonlinear Imaging Models using Deep Generative Priors](https://mlsanity.com/paper/2203.10078/) (2022)
- [Bayesian Posterior Perturbation Analysis with Integral Probability Metrics](https://mlsanity.com/paper/2303.01512/) (2023)
- [Inverse problem regularization with hierarchical variational autoencoders](https://mlsanity.com/paper/2303.11217/) (2023)
- [Do Bayesian imaging methods report trustworthy probabilities?](https://mlsanity.com/paper/2405.08179/) (2024)
- [Paired Autoencoders for Likelihood-free Estimation in Inverse Problems](https://mlsanity.com/paper/2405.13220/) (2024)
- [Good Things Come in Pairs: Paired Autoencoders for Inverse Problems](https://mlsanity.com/paper/2505.06549/) (2025)
- [Variational Bayes image restoration with compressive autoencoders](https://mlsanity.com/paper/2311.17744/) (2023)
- [Normalization-equivariant Diffusion Models: Learning Posterior Samplers From Noisy And Partial Measurements](https://mlsanity.com/paper/2510.11964/) (2025)
- [Estimating a potential without the agony of the partition function](https://mlsanity.com/paper/2208.09433/) (2022)
- [Generative models and Bayesian inversion using Laplace approximation](https://mlsanity.com/paper/2203.07755/) (2022)
- [Robustness and Exploration of Variational and Machine Learning Approaches to Inverse Problems: An Overview](https://mlsanity.com/paper/2402.12072/) (2024)
- [Uniform Transformation: Refining Latent Representation in Variational Autoencoders](https://mlsanity.com/paper/2407.02681/) (2024)
- [Deep priors for satellite image restoration with accurate uncertainties](https://mlsanity.com/paper/2412.04130/) (2024)
- [Diverse super-resolution with pretrained deep hiererarchical VAEs](https://mlsanity.com/paper/2205.10347/) (2022)
- [Bayesian imaging inverse problem with SA-Roundtrip prior via HMC-pCN sampler](https://mlsanity.com/paper/2310.17817/) (2023)
- [A Multi-stage Constrained Optimization Framework for Data-driven Problems](https://mlsanity.com/paper/2607.23480/) (2026)

## Abstract (arXiv)

In this work we address the problem of solving ill-posed inverse problems in imaging where the prior is a variational autoencoder (VAE). Specifically we consider the decoupled case where the prior is trained once and can be reused for many different log-concave degradation models without retraining. Whereas previous MAP-based approaches to this problem lead to highly non-convex optimization algorithms, our approach computes the joint (space-latent) MAP that naturally leads to alternate optimization algorithms and to the use of a stochastic encoder to accelerate computations. The resulting technique (JPMAP) performs Joint Posterior Maximization using an Autoencoding Prior. We show theoretical and experimental evidence that the proposed objective function is quite close to bi-convex. Indeed it satisfies a weak bi-convexity property which is sufficient to guarantee that our optimization scheme converges to a stationary point. We also highlight the importance of correctly training the VAE using a denoising criterion, in order to ensure that the encoder generalizes well to out-of-distribution images, without affecting the quality of the generative model. This simple modification is key to providing robustness to the whole procedure. Finally we show how our joint MAP methodology relates to more common MAP approaches, and we propose a continuation scheme that makes use of our JPMAP algorithm to provide more robust MAP estimates. Experimental results also show the higher quality of the solutions obtained by our JPMAP approach with respect to other non-convex MAP approaches which more often get stuck in spurious local optima.