# Gradient Variance Loss for Structure-Enhanced Image Super-Resolution

arXiv: https://arxiv.org/abs/2202.00997 · HTML: https://mlsanity.com/paper/2202.00997/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 10 of 10)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis](https://mlsanity.com/paper/1612.07919/) (2016)
- [Structure-Preserving Super Resolution with Gradient Guidance](https://mlsanity.com/paper/2003.13081/) (2020)

## Built on by (most cited first, 12 of 12)

- [NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results](https://mlsanity.com/paper/2205.05675/) (2022)
- [The Tenth NTIRE 2025 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2504.10686/) (2025)
- [Edge-enhanced Feature Distillation Network for Efficient Super-Resolution](https://mlsanity.com/paper/2204.08759/) (2022)
- [Data-Driven Filter Design in FBP: Transforming CT Reconstruction with Trainable Fourier Series](https://mlsanity.com/paper/2401.16039/) (2024)
- [EAGLE: An Edge-Aware Gradient Localization Enhanced Loss for CT Image Reconstruction](https://mlsanity.com/paper/2403.10695/) (2024)
- [One-Shot Face Video Re-enactment using Hybrid Latent Spaces of StyleGAN2](https://mlsanity.com/paper/2302.07848/) (2023)
- [Your Super Resolution Model is not Enough for Tackling Real-World Scenarios](https://mlsanity.com/paper/2509.06387/) (2025)
- [Trainable Loss Weights in Super-Resolution](https://mlsanity.com/paper/2301.10575/) (2023)
- [Super-Resolution Generative Adversarial Networks based Video Enhancement](https://mlsanity.com/paper/2505.10589/) (2025)
- [Enhancing Frequency for Single Image Super-Resolution with Learnable Separable Kernels](https://mlsanity.com/paper/2506.04555/) (2025)
- [Efficient Perceptual Image Super Resolution: AIM 2025 Study and Benchmark](https://mlsanity.com/paper/2510.12765/) (2025)
- [GrainSpeech: Less Context, More Detail for Compact Speech Synthesis](https://mlsanity.com/paper/2609.18856/) (2026)

## Abstract (arXiv)

Recent success in the field of single image super-resolution (SISR) is achieved by optimizing deep convolutional neural networks (CNNs) in the image space with the L1 or L2 loss. However, when trained with these loss functions, models usually fail to recover sharp edges present in the high-resolution (HR) images for the reason that the model tends to give a statistical average of potential HR solutions. During our research, we observe that gradient maps of images generated by the models trained with the L1 or L2 loss have significantly lower variance than the gradient maps of the original high-resolution images. In this work, we propose to alleviate the above issue by introducing a structure-enhancing loss function, coined Gradient Variance (GV) loss, and generate textures with perceptual-pleasant details. Specifically, during the training of the model, we extract patches from the gradient maps of the target and generated output, calculate the variance of each patch and form variance maps for these two images. Further, we minimize the distance between the computed variance maps to enforce the model to produce high variance gradient maps that will lead to the generation of high-resolution images with sharper edges. Experimental results show that the GV loss can significantly improve both Structure Similarity (SSIM) and peak signal-to-noise ratio (PSNR) performance of existing image super-resolution (SR) deep learning models.