# Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image Restoration

arXiv: https://arxiv.org/abs/2306.02342 · HTML: https://mlsanity.com/paper/2306.02342/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 27 of 27)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Optimal transport mapping via input convex neural networks](https://mlsanity.com/paper/1908.10962/) (2019)
- [SNIPS: Solving Noisy Inverse Problems Stochastically](https://mlsanity.com/paper/2105.14951/) (2021)
- [Neural Optimal Transport](https://mlsanity.com/paper/2201.12220/) (2022)
- [Wasserstein-2 Generative Networks](https://mlsanity.com/paper/1909.13082/) (2019)
- [Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration](https://mlsanity.com/paper/2209.11345/) (2022)
- [A Theory of the Distortion-Perception Tradeoff in Wasserstein Space](https://mlsanity.com/paper/2107.02555/) (2021)
- [Stochastic Image Denoising by Sampling from the Posterior Distribution](https://mlsanity.com/paper/2101.09552/) (2021)
- [A Closed-form Solution to Universal Style Transfer](https://mlsanity.com/paper/1906.00668/) (2019)
- [High Perceptual Quality Image Denoising with a Posterior Sampling CGAN](https://mlsanity.com/paper/2103.04192/) (2021)
- [Wasserstein Style Transfer](https://mlsanity.com/paper/1905.12828/) (2019)
- [Reasons for the Superiority of Stochastic Estimators over Deterministic Ones: Robustness, Consistency and Perceptual Quality](https://mlsanity.com/paper/2211.08944/) (2022)

## Built on by (most cited first, 15 of 15)

- [Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration](https://mlsanity.com/paper/2410.00418/) (2024)
- [Degradation-Aware Residual-Conditioned Optimal Transport for Unified Image Restoration](https://mlsanity.com/paper/2411.01656/) (2024)
- [Taming Diffusion Models for Image Restoration: A Review](https://mlsanity.com/paper/2409.10353/) (2024)
- [Enhanced Control for Diffusion Bridge in Image Restoration](https://mlsanity.com/paper/2408.16303/) (2024)
- [Domain-Aware Few-Shot Learning for Optical Coherence Tomography Noise Reduction](https://mlsanity.com/paper/2306.08102/) (2023)
- [An End-to-End Real-World Camera Imaging Pipeline](https://mlsanity.com/paper/2411.10773/) (2024)
- [Perceptual Fairness in Image Restoration](https://mlsanity.com/paper/2405.13805/) (2024)
- [Constrained Sliced Wasserstein Embedding](https://mlsanity.com/paper/2506.02203/) (2025)
- [Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward](https://mlsanity.com/paper/2511.01645/) (2025)
- [Characterization of the Distortion-Perception Tradeoff for Finite Channels with Arbitrary Metrics](https://mlsanity.com/paper/2402.02265/) (2024)
- [A Simple Combination of Diffusion Models for Better Quality Trade-Offs in Image Denoising](https://mlsanity.com/paper/2503.14654/) (2025)
- [Regularized Schr"odinger Bridge via Distortion-Perception Perturbation for High-Fidelity Speech Enhancement](https://mlsanity.com/paper/2511.11686/) (2025)
- [Flow Straight to Reality: Perceptually Consistent Flow Matching for Efficient Image Restoration](https://mlsanity.com/paper/2608.10544/) (2026)
- [Mean and Variance Estimation Complexity in Arbitrary Distributions via Wasserstein Minimization](https://mlsanity.com/paper/2501.10172/) (2025)
- [Test-Time Adaptation for Anomaly Segmentation via Topology-Aware Optimal Transport Chaining](https://mlsanity.com/paper/2601.20333/) (2026)

## Abstract (arXiv)

We propose an image restoration algorithm that can control the perceptual quality and/or the mean square error (MSE) of any pre-trained model, trading one over the other at test time. Our algorithm is few-shot: Given about a dozen images restored by the model, it can significantly improve the perceptual quality and/or the MSE of the model for newly restored images without further training. Our approach is motivated by a recent theoretical result that links between the minimum MSE (MMSE) predictor and the predictor that minimizes the MSE under a perfect perceptual quality constraint. Specifically, it has been shown that the latter can be obtained by optimally transporting the output of the former, such that its distribution matches the source data. Thus, to improve the perceptual quality of a predictor that was originally trained to minimize MSE, we approximate the optimal transport by a linear transformation in the latent space of a variational auto-encoder, which we compute in closed-form using empirical means and covariances. Going beyond the theory, we find that applying the same procedure on models that were initially trained to achieve high perceptual quality, typically improves their perceptual quality even further. And by interpolating the results with the original output of the model, we can improve their MSE on the expense of perceptual quality. We illustrate our method on a variety of degradations applied to general content images of arbitrary dimensions.