# Residual-Conditioned Optimal Transport: Towards Structure-Preserving Unpaired and Paired Image Restoration

arXiv: https://arxiv.org/abs/2405.02843 · HTML: https://mlsanity.com/paper/2405.02843/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 65 references that are in the map.
- First posted on arXiv 5 May 2024.

## Builds on (most cited first, 30 of 39)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Benchmarking Single Image Dehazing and Beyond](https://mlsanity.com/paper/1712.04143/) (2017)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Denoising Diffusion Models for Plug-and-Play Image Restoration](https://mlsanity.com/paper/2305.08995/) (2023)
- [Vision Transformers for Single Image Dehazing](https://mlsanity.com/paper/2204.03883/) (2022)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [Learning A Sparse Transformer Network for Effective Image Deraining](https://mlsanity.com/paper/2303.11950/) (2023)
- [Image De-raining Using a Conditional Generative Adversarial Network](https://mlsanity.com/paper/1701.05957/) (2017)
- [Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset](https://mlsanity.com/paper/1904.01538/) (2019)
- [SNIPS: Solving Noisy Inverse Problems Stochastically](https://mlsanity.com/paper/2105.14951/) (2021)
- [Image Restoration with Mean-Reverting Stochastic Differential Equations](https://mlsanity.com/paper/2301.11699/) (2023)
- [RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution](https://mlsanity.com/paper/1908.06382/) (2019)
- [Neural Optimal Transport](https://mlsanity.com/paper/2201.12220/) (2022)
- [PromptIR: Prompting for All-in-One Blind Image Restoration](https://mlsanity.com/paper/2306.13090/) (2023)
- [O-HAZE: a dehazing benchmark with real hazy and haze-free outdoor images](https://mlsanity.com/paper/1804.05101/) (2018)
- [NAFSSR: Stereo Image Super-Resolution Using NAFNet](https://mlsanity.com/paper/2204.08714/) (2022)
- [A General Decoupled Learning Framework for Parameterized Image Operators](https://mlsanity.com/paper/1907.05852/) (2019)

## Built on by (most cited first, 13 of 13)

- [Degradation-Aware Residual-Conditioned Optimal Transport for Unified Image Restoration](https://mlsanity.com/paper/2411.01656/) (2024)
- [Joint Velocity-Growth Flow Matching for Single-Cell Dynamics Modeling](https://mlsanity.com/paper/2505.13413/) (2025)
- [Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration](https://mlsanity.com/paper/2602.23169/) (2026)
- [Universal Image Restoration Pre-training via Masked Degradation Classification](https://mlsanity.com/paper/2510.13282/) (2025)
- [Adaptive Blind All-in-One Image Restoration](https://mlsanity.com/paper/2411.18412/) (2024)
- [BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restoration](https://mlsanity.com/paper/2505.21637/) (2025)
- [Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward](https://mlsanity.com/paper/2511.01645/) (2025)
- [Towards Prospective Medical Image Reconstruction via Knowledge-Informed Dynamic Optimal Transport](https://mlsanity.com/paper/2505.17644/) (2025)
- [WeatherCycle: Unpaired Multi-Weather Restoration via Color Space Decoupled Cycle Learning](https://mlsanity.com/paper/2509.23150/) (2025)
- [Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization](https://mlsanity.com/paper/2410.02628/) (2024)
- [Generative Latent Kernel Modeling for Blind Motion Deblurring](https://mlsanity.com/paper/2507.09285/) (2025)
- [Bridging Degradation Discrimination and Generation for Universal Image Restoration](https://mlsanity.com/paper/2602.00579/) (2026)
- [UOTIP: Unbalanced Optimal Transport Map for Unpaired Inverse Problems](https://mlsanity.com/paper/2605.21094/) (2026)

## Abstract (arXiv)

Deep learning-based image restoration methods generally struggle with faithfully preserving the structures of the original image. In this work, we propose a novel Residual-Conditioned Optimal Transport (RCOT) approach, which models image restoration as an optimal transport (OT) problem for both unpaired and paired settings, introducing the transport residual as a unique degradation-specific cue for both the transport cost and the transport map. Specifically, we first formalize a Fourier residual-guided OT objective by incorporating the degradation-specific information of the residual into the transport cost. We further design the transport map as a two-pass RCOT map that comprises a base model and a refinement process, in which the transport residual is computed by the base model in the first pass and then encoded as a degradation-specific embedding to condition the second-pass restoration. By duality, the RCOT problem is transformed into a minimax optimization problem, which can be solved by adversarially training neural networks. Extensive experiments on multiple restoration tasks show that RCOT achieves competitive performance in terms of both distortion measures and perceptual quality, restoring images with more faithful structures as compared with state-of-the-art methods.