# Learning to Generate Images with Perceptual Similarity Metrics

arXiv: https://arxiv.org/abs/1511.06409 · HTML: https://mlsanity.com/paper/1511.06409/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2015.

## Built on by (most cited first, 30 of 39)

- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case](https://mlsanity.com/paper/2206.08309/) (2022)
- [Robust Representation Learning via Perceptual Similarity Metrics](https://mlsanity.com/paper/2106.06620/) (2021)
- [UAV Localization Using Autoencoded Satellite Images](https://mlsanity.com/paper/2102.05692/) (2021)
- [Lifting Architectural Constraints of Injective Flows](https://mlsanity.com/paper/2306.01843/) (2023)
- [Person Re-Identification without Identification via Event Anonymization](https://mlsanity.com/paper/2308.04402/) (2023)
- [Residual Diffusion Bridge Model for Image Restoration](https://mlsanity.com/paper/2510.23116/) (2025)
- [A Systematic Performance Analysis of Deep Perceptual Loss Networks: Breaking Transfer Learning Conventions](https://mlsanity.com/paper/2302.04032/) (2023)
- [DC-cycleGAN: Bidirectional CT-to-MR Synthesis from Unpaired Data](https://mlsanity.com/paper/2211.01293/) (2022)
- [A robust and scalable framework for hallucination detection in virtual tissue staining and digital pathology](https://mlsanity.com/paper/2404.18458/) (2024)
- [Scalable Font Reconstruction with Dual Latent Manifolds](https://mlsanity.com/paper/2109.06627/) (2021)
- [Can we integrate spatial verification methods into neural-network loss functions for atmospheric science?](https://mlsanity.com/paper/2203.11141/) (2022)
- [Neural Neural Textures Make Sim2Real Consistent](https://mlsanity.com/paper/2206.13500/) (2022)
- [Inductive biases in deep learning models for weather prediction](https://mlsanity.com/paper/2304.04664/) (2023)
- [Alt-Text with Context: Improving Accessibility for Images on Twitter](https://mlsanity.com/paper/2305.14779/) (2023)
- [IQPFR: An Image Quality Prior for Blind Face Restoration and Beyond](https://mlsanity.com/paper/2503.09294/) (2025)
- [SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling](https://mlsanity.com/paper/2506.21041/) (2025)
- [CAGAN: Text-To-Image Generation with Combined Attention GANs](https://mlsanity.com/paper/2104.12663/) (2021)
- [Evaluating the Quality and Diversity of DCGAN-based Generatively Synthesized Diabetic Retinopathy Imagery](https://mlsanity.com/paper/2208.05593/) (2022)
- [CCR: Facial Image Editing with Continuity, Consistency and Reversibility](https://mlsanity.com/paper/2209.10734/) (2022)
- [An Attentive-based Generative Model for Medical Image Synthesis](https://mlsanity.com/paper/2306.01562/) (2023)
- [CSIM: A Copula-based similarity index sensitive to local changes for Image quality assessment](https://mlsanity.com/paper/2410.01411/) (2024)
- [Toward Theoretical Insights into Diffusion Trajectory Distillation via Operator Merging](https://mlsanity.com/paper/2505.16024/) (2025)
- [DriveGuard: Robustification of Automated Driving Systems with Deep Spatio-Temporal Convolutional Autoencoder](https://mlsanity.com/paper/2111.03480/) (2021)
- [Pix2HDR -- A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos](https://mlsanity.com/paper/2310.16139/) (2023)
- [Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning](https://mlsanity.com/paper/2311.02815/) (2023)
- [Enhance the Image: Super Resolution using Artificial Intelligence in MRI](https://mlsanity.com/paper/2406.13625/) (2024)
- [Face Photo-Sketch Recognition Using Bidirectional Collaborative Synthesis Network](https://mlsanity.com/paper/2108.09898/) (2021)
- [Face Reconstruction with Variational Autoencoder and Face Masks](https://mlsanity.com/paper/2112.02139/) (2021)

## Abstract (arXiv)

Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM). Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training deterministic and stochastic autoencoders. For three different architectures, we collected human judgments of the quality of image reconstructions. Observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures (ℓ1 and ℓ2 distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. Just as computer vision has advanced through the use of convolutional architectures that mimic the structure of the mammalian visual system, we argue that significant additional advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.