# Generic Perceptual Loss for Modeling Structured Output Dependencies

arXiv: https://arxiv.org/abs/2103.10571 · HTML: https://mlsanity.com/paper/2103.10571/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [Learning to Recover 3D Scene Shape from a Single Image](https://mlsanity.com/paper/2012.09365/) (2020)
- [BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation](https://mlsanity.com/paper/2001.00309/) (2020)
- [Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models](https://mlsanity.com/paper/1808.01688/) (2018)
- [Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks](https://mlsanity.com/paper/1605.02305/) (2016)
- [Efficient piecewise training of deep structured models for semantic segmentation](https://mlsanity.com/paper/1504.01013/) (2015)
- [Weight Agnostic Neural Networks](https://mlsanity.com/paper/1906.04358/) (2019)
- [SOLOv2: Dynamic and Fast Instance Segmentation](https://mlsanity.com/paper/2003.10152/) (2020)
- [Unsupervised Representation Learning by Predicting Random Distances](https://mlsanity.com/paper/1912.12186/) (2019)
- [Task-Aware Monocular Depth Estimation for 3D Object Detection](https://mlsanity.com/paper/1909.07701/) (2019)
- [Representative Graph Neural Network](https://mlsanity.com/paper/2008.05202/) (2020)

## Built on by (most cited first, 9 of 9)

- [Residual Local Feature Network for Efficient Super-Resolution](https://mlsanity.com/paper/2205.07514/) (2022)
- [NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results](https://mlsanity.com/paper/2205.05675/) (2022)
- [ViT-AE++: Improving Vision Transformer Autoencoder for Self-supervised Medical Image Representations](https://mlsanity.com/paper/2301.07382/) (2023)
- [Cyclic 2.5D Perceptual Loss for Cross-Modal 3D Medical Image Synthesis: T1w MRI to Tau PET](https://mlsanity.com/paper/2406.12632/) (2024)
- [FREDSR: Fourier Residual Efficient Diffusive GAN for Single Image Super Resolution](https://mlsanity.com/paper/2211.16678/) (2022)
- [A Domain-specific Perceptual Metric via Contrastive Self-supervised Representation: Applications on Natural and Medical Images](https://mlsanity.com/paper/2212.01577/) (2022)
- [Adversarial Purification of Information Masking](https://mlsanity.com/paper/2311.15339/) (2023)
- [Untrained Perceptual Loss for image denoising of line-like structures in MR images](https://mlsanity.com/paper/2411.05884/) (2024)
- [Map Feature Perception Metric for Map Generation Quality Assessment and Loss Optimization](https://mlsanity.com/paper/2503.23370/) (2025)

## Abstract (arXiv)

The perceptual loss has been widely used as an effective loss term in image synthesis tasks including image super-resolution, and style transfer. It was believed that the success lies in the high-level perceptual feature representations extracted from CNNs pretrained with a large set of images. Here we reveal that, what matters is the network structure instead of the trained weights. Without any learning, the structure of a deep network is sufficient to capture the dependencies between multiple levels of variable statistics using multiple layers of CNNs. This insight removes the requirements of pre-training and a particular network structure (commonly, VGG) that are previously assumed for the perceptual loss, thus enabling a significantly wider range of applications. To this end, we demonstrate that a randomly-weighted deep CNN can be used to model the structured dependencies of outputs. On a few dense per-pixel prediction tasks such as semantic segmentation, depth estimation and instance segmentation, we show improved results of using the extended randomized perceptual loss, compared to the baselines using pixel-wise loss alone. We hope that this simple, extended perceptual loss may serve as a generic structured-output loss that is applicable to most structured output learning tasks.