# HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement in Game Engines

arXiv: https://arxiv.org/abs/2603.10604 · HTML: https://mlsanity.com/paper/2603.10604/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 30 of 35)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Learning from Synthetic Data for Crowd Counting in the Wild](https://mlsanity.com/paper/1903.03303/) (2019)
- [A U-Net Based Discriminator for Generative Adversarial Networks](https://mlsanity.com/paper/2002.12655/) (2020)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [Understanding Hallucinations in Diffusion Models through Mode Interpolation](https://mlsanity.com/paper/2406.09358/) (2024)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [Dual Contrastive Learning for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2104.07689/) (2021)
- [Spatially-Adaptive Pixelwise Networks for Fast Image Translation](https://mlsanity.com/paper/2012.02992/) (2020)
- [Contrastive Language-Image Pre-Training with Knowledge Graphs](https://mlsanity.com/paper/2210.08901/) (2022)
- [Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics](https://mlsanity.com/paper/2012.04932/) (2020)
- [Unpaired Image Translation via Vector Symbolic Architectures](https://mlsanity.com/paper/2209.02686/) (2022)
- [CARLA2Real: a tool for reducing the sim2real appearance gap in CARLA simulator](https://mlsanity.com/paper/2410.18238/) (2024)
- [Optical Flow Dataset and Benchmark for Visual Crowd Analysis](https://mlsanity.com/paper/1811.07170/) (2018)

## Built on by (most cited first, 1 of 1)

- [The Domain Is a Residue: Adapting Self-Supervised Features, Not Generators](https://mlsanity.com/paper/2609.37330/) (2026)

## Abstract (arXiv)

Generative models are increasingly used in video game engines to enhance the photorealism of rendered images for visual synthetic data generation and simulation applications. However, they often introduce artifacts that alter the content of the original rendered scenes and require high computational resources, which limit their utilization for the photorealism enhancement of training and evaluation data, as well as their integration in the rendering pipelines of game engines. In this paper, we propose Hybrid Patch Enhanced Realism Generative Adversarial Network (HyPER-GAN), a hybrid image-to-image translation framework that is based on a lightweight U-Net-style generator capable of performing real-time inference. The framework is trained using paired rendered and photorealism-enhanced images, complemented by a novel hybrid training strategy that incorporates matched patches from unpaired real-world images to improve content preservation and further enhance the visual realism that can be achieved by the lightweight generator. Experimental results demonstrate that HyPER-GAN achieves a 6x increase in frames per second at 1080p in comparison with state-of-the-art lightweight paired image-to-image translation methods, while also increasing, in both within- and cross-engine evaluations, the photorealism of the rendered images without significantly compromising semantic consistency. Moreover, it is illustrated that HyPER-GAN maintains temporal consistency and that the proposed hybrid training strategy improves content preservation and visual realism in within-engine and increases the robustness in cross-engine evaluations compared to training the framework solely with paired rendered and photorealism-enhanced images. Code and pretrained models are publicly available at: https://github.com/stefanos50/HyPER-GAN