# Training a Task-Specific Image Reconstruction Loss

arXiv: https://arxiv.org/abs/2103.14616 · HTML: https://mlsanity.com/paper/2103.14616/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks](https://mlsanity.com/paper/1506.05751/) (2015)
- [EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis](https://mlsanity.com/paper/1612.07919/) (2016)
- [DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks](https://mlsanity.com/paper/1704.02470/) (2017)
- [Compression Artifacts Reduction by a Deep Convolutional Network](https://mlsanity.com/paper/1504.06993/) (2015)
- [Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis](https://mlsanity.com/paper/1601.04589/) (2016)
- [Explorable Super Resolution](https://mlsanity.com/paper/1912.01839/) (2019)
- [Deep Generative Adversarial Compression Artifact Removal](https://mlsanity.com/paper/1704.02518/) (2017)
- [Projected Distribution Loss for Image Enhancement](https://mlsanity.com/paper/2012.09289/) (2020)
- [Transformation Consistency Regularization- A Semi-Supervised Paradigm for Image-to-Image Translation](https://mlsanity.com/paper/2007.07867/) (2020)
- [A practical guide and software for analysing pairwise comparison experiments](https://mlsanity.com/paper/1712.03686/) (2017)
- [Enhancing Perceptual Loss with Adversarial Feature Matching for Super-Resolution](https://mlsanity.com/paper/2005.07502/) (2020)

## Built on by (most cited first, 16 of 16)

- [OMGSR: You Only Need One Mid-timestep Guidance for Real-World Image Super-Resolution](https://mlsanity.com/paper/2508.08227/) (2025)
- [Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks](https://mlsanity.com/paper/2310.06958/) (2023)
- [MuS2: A Real-World Benchmark for Sentinel-2 Multi-Image Super-Resolution](https://mlsanity.com/paper/2210.02745/) (2022)
- [IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding](https://mlsanity.com/paper/2508.09456/) (2025)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning](https://mlsanity.com/paper/2602.03390/) (2026)
- [Tunable Convolutions with Parametric Multi-Loss Optimization](https://mlsanity.com/paper/2304.00898/) (2023)
- [Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning](https://mlsanity.com/paper/2603.22758/) (2026)
- [Enhancing Risk Assessment in Transformers with Loss-at-Risk Functions](https://mlsanity.com/paper/2411.02558/) (2024)
- [ISP meets Deep Learning: A Survey on Deep Learning Methods for Image Signal Processing](https://mlsanity.com/paper/2305.11994/) (2023)
- [LuminanceL1Loss: A loss function which measures percieved brightness and colour differences](https://mlsanity.com/paper/2311.04614/) (2023)
- [Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach](https://mlsanity.com/paper/2404.05768/) (2024)
- [StarryGazer: Leveraging Monocular Depth Estimation Models for Domain-Agnostic Single Depth Image Completion](https://mlsanity.com/paper/2512.13147/) (2025)
- [Direct Unsupervised Denoising](https://mlsanity.com/paper/2310.18116/) (2023)
- [Map Feature Perception Metric for Map Generation Quality Assessment and Loss Optimization](https://mlsanity.com/paper/2503.23370/) (2025)
- [Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Data](https://mlsanity.com/paper/2604.02031/) (2026)

## Abstract (arXiv)

The choice of a loss function is an important factor when training neural networks for image restoration problems, such as single image super resolution. The loss function should encourage natural and perceptually pleasing results. A popular choice for a loss is a pre-trained network, such as VGG, which is used as a feature extractor for computing the difference between restored and reference images. However, such an approach has multiple drawbacks: it is computationally expensive, requires regularization and hyper-parameter tuning, and involves a large network trained on an unrelated task. Furthermore, it has been observed that there is no single loss function that works best across all applications and across different datasets. In this work, we instead propose to train a set of loss functions that are application specific in nature. Our loss function comprises a series of discriminators that are trained to detect and penalize the presence of application-specific artifacts. We show that a single natural image and corresponding distortions are sufficient to train our feature extractor that outperforms state-of-the-art loss functions in applications like single image super resolution, denoising, and JPEG artifact removal. Finally, we conclude that an effective loss function does not have to be a good predictor of perceived image quality, but instead needs to be specialized in identifying the distortions for a given restoration method.