# Neural-Polyptych: Content Controllable Painting Recreation for Diverse Genres

arXiv: https://arxiv.org/abs/2409.19690 · HTML: https://mlsanity.com/paper/2409.19690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 52 references that are in the map.
- First posted on arXiv 29 Sep 2024.

## Builds on (most cited first, 30 of 36)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning](https://mlsanity.com/paper/1705.07761/) (2017)
- [Neural Style Transfer: A Review](https://mlsanity.com/paper/1705.04058/) (2017)
- [Demystifying Neural Style Transfer](https://mlsanity.com/paper/1701.01036/) (2017)
- [Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis](https://mlsanity.com/paper/2101.04775/) (2021)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Multi-Content GAN for Few-Shot Font Style Transfer](https://mlsanity.com/paper/1712.00516/) (2017)
- [Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2205.09542/) (2022)
- [Aligning Latent and Image Spaces to Connect the Unconnectable](https://mlsanity.com/paper/2104.06954/) (2021)

## Abstract (arXiv)

To bridge the gap between artists and non-specialists, we present a unified framework, Neural-Polyptych, to facilitate the creation of expansive, high-resolution paintings by seamlessly incorporating interactive hand-drawn sketches with fragments from original paintings. We have designed a multi-scale GAN-based architecture to decompose the generation process into two parts, each responsible for identifying global and local features. To enhance the fidelity of semantic details generated from users' sketched outlines, we introduce a Correspondence Attention module utilizing our Reference Bank strategy. This ensures the creation of high-quality, intricately detailed elements within the artwork. The final result is achieved by carefully blending these local elements while preserving coherent global consistency. Consequently, this methodology enables the production of digital paintings at megapixel scale, accommodating diverse artistic expressions and enabling users to recreate content in a controlled manner. We validate our approach to diverse genres of both Eastern and Western paintings. Applications such as large painting extension, texture shuffling, genre switching, mural art restoration, and recomposition can be successfully based on our framework.