# Perceptual-Distortion Balanced Image Super-Resolution is a Multi-Objective Optimization Problem

arXiv: https://arxiv.org/abs/2409.03179 · HTML: https://mlsanity.com/paper/2409.03179/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 61 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 30 of 32)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Accelerating the Super-Resolution Convolutional Neural Network](https://mlsanity.com/paper/1608.00367/) (2016)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Sketch-based Manga Retrieval using Manga109 Dataset](https://mlsanity.com/paper/1510.04389/) (2015)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis](https://mlsanity.com/paper/1612.07919/) (2016)
- [The 2018 PIRM Challenge on Perceptual Image Super-resolution](https://mlsanity.com/paper/1809.07517/) (2018)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [Structure-Preserving Super Resolution with Gradient Guidance](https://mlsanity.com/paper/2003.13081/) (2020)
- [Frequency Separation for Real-World Super-Resolution](https://mlsanity.com/paper/1911.07850/) (2019)
- [Fourier Space Losses for Efficient Perceptual Image Super-Resolution](https://mlsanity.com/paper/2106.00783/) (2021)
- [Deep Network Interpolation for Continuous Imagery Effect Transition](https://mlsanity.com/paper/1811.10515/) (2018)
- [ShuffleMixer: An Efficient ConvNet for Image Super-Resolution](https://mlsanity.com/paper/2205.15175/) (2022)
- [SROBB: Targeted Perceptual Loss for Single Image Super-Resolution](https://mlsanity.com/paper/1908.07222/) (2019)
- [Wavelet Domain Style Transfer for an Effective Perception-distortion Tradeoff in Single Image Super-Resolution](https://mlsanity.com/paper/1910.04074/) (2019)
- [CFSNet: Toward a Controllable Feature Space for Image Restoration](https://mlsanity.com/paper/1904.00634/) (2019)
- [Perception-Oriented Single Image Super-Resolution using Optimal Objective Estimation](https://mlsanity.com/paper/2211.13676/) (2022)
- [Edge-enhanced Feature Distillation Network for Efficient Super-Resolution](https://mlsanity.com/paper/2204.08759/) (2022)

## Built on by (most cited first, 3 of 3)

- [XYScanNet: A State Space Model for Single Image Deblurring](https://mlsanity.com/paper/2412.10338/) (2024)
- [CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution](https://mlsanity.com/paper/2503.14272/) (2025)
- [Revisiting the Perception-Distortion Trade-off with Spatial-Semantic Guided Super-Resolution](https://mlsanity.com/paper/2603.14112/) (2026)

## Abstract (arXiv)

Training Single-Image Super-Resolution (SISR) models using pixel-based regression losses can achieve high distortion metrics scores (e.g., PSNR and SSIM), but often results in blurry images due to insufficient recovery of high-frequency details. Conversely, using GAN or perceptual losses can produce sharp images with high perceptual metric scores (e.g., LPIPS), but may introduce artifacts and incorrect textures. Balancing these two types of losses can help achieve a trade-off between distortion and perception, but the challenge lies in tuning the loss function weights. To address this issue, we propose a novel method that incorporates Multi-Objective Optimization (MOO) into the training process of SISR models to balance perceptual quality and distortion. We conceptualize the relationship between loss weights and image quality assessment (IQA) metrics as black-box objective functions to be optimized within our Multi-Objective Bayesian Optimization Super-Resolution (MOBOSR) framework. This approach automates the hyperparameter tuning process, reduces overall computational cost, and enables the use of numerous loss functions simultaneously. Extensive experiments demonstrate that MOBOSR outperforms state-of-the-art methods in terms of both perceptual quality and distortion, significantly advancing the perception-distortion Pareto frontier. Our work points towards a new direction for future research on balancing perceptual quality and fidelity in nearly all image restoration tasks. The source code and pretrained models are available at: https://github.com/ZhuKeven/MOBOSR.