# Generative Adversarial Shaders for Real-Time Realism Enhancement

arXiv: https://arxiv.org/abs/2306.04629 · HTML: https://mlsanity.com/paper/2306.04629/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 27 of 27)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [EnlightenGAN: Deep Light Enhancement without Paired Supervision](https://mlsanity.com/paper/1906.06972/) (2019)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks](https://mlsanity.com/paper/1604.04382/) (2016)
- [Photorealistic Style Transfer via Wavelet Transforms](https://mlsanity.com/paper/1903.09760/) (2019)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [Amortised MAP Inference for Image Super-resolution](https://mlsanity.com/paper/1610.04490/) (2016)
- [TSIT: A Simple and Versatile Framework for Image-to-Image Translation](https://mlsanity.com/paper/2007.12072/) (2020)
- [Learning Linear Transformations for Fast Arbitrary Style Transfer](https://mlsanity.com/paper/1808.04537/) (2018)
- [Deep Shading: Convolutional Neural Networks for Screen-Space Shading](https://mlsanity.com/paper/1603.06078/) (2016)
- [UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translation](https://mlsanity.com/paper/2203.02557/) (2022)
- [Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation](https://mlsanity.com/paper/1809.06256/) (2018)
- [Deep Illumination: Approximating Dynamic Global Illumination with Generative Adversarial Network](https://mlsanity.com/paper/1710.09834/) (2017)

## Built on by (most cited first, 1 of 1)

- [Fast Local Neural Regression for Low-Cost, Path Traced Lambertian Global Illumination](https://mlsanity.com/paper/2410.11625/) (2024)

## Abstract (arXiv)

Application of realism enhancement methods, particularly in real-time and resource-constrained settings, has been frustrated by the expense of existing methods. These achieve high quality results only at the cost of long runtimes and high bandwidth, memory, and power requirements. We present an efficient alternative: a high-performance, generative shader-based approach that adapts machine learning techniques to real-time applications, even in resource-constrained settings such as embedded and mobile GPUs. The proposed learnable shader pipeline comprises differentiable functions that can be trained in an end-to-end manner using an adversarial objective, allowing for faithful reproduction of the appearance of a target image set without manual tuning. The shader pipeline is optimized for highly efficient execution on the target device, providing temporally stable, faster-than-real time results with quality competitive with many neural network-based methods.