# FISTNet: FusIon of STyle-path generative Networks for Facial Style Transfer

arXiv: https://arxiv.org/abs/2307.09020 · HTML: https://mlsanity.com/paper/2307.09020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Few-shot Image Generation via Cross-domain Correspondence](https://mlsanity.com/paper/2104.06820/) (2021)
- [Visual Attribute Transfer through Deep Image Analogy](https://mlsanity.com/paper/1705.01088/) (2017)
- [Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis](https://mlsanity.com/paper/1601.04589/) (2016)
- [Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains](https://mlsanity.com/paper/2010.05334/) (2020)
- [BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation](https://mlsanity.com/paper/2110.11728/) (2021)
- [Unpaired Image-to-Image Translation using Adversarial Consistency Loss](https://mlsanity.com/paper/2003.04858/) (2020)
- [Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network](https://mlsanity.com/paper/2010.05713/) (2020)
- [WarpGAN: Automatic Caricature Generation](https://mlsanity.com/paper/1811.10100/) (2018)
- [Face Generation and Editing with StyleGAN: A Survey](https://mlsanity.com/paper/2212.09102/) (2022)
- [AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation](https://mlsanity.com/paper/2102.12593/) (2021)
- [VToonify: Controllable High-Resolution Portrait Video Style Transfer](https://mlsanity.com/paper/2209.11224/) (2022)
- [SPatchGAN: A Statistical Feature Based Discriminator for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2103.16219/) (2021)
- [GAN-based Facial Attribute Manipulation](https://mlsanity.com/paper/2210.12683/) (2022)
- [GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too!)](https://mlsanity.com/paper/2106.06561/) (2021)
- [MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing](https://mlsanity.com/paper/2004.10634/) (2020)
- [FaceTuneGAN: Face Autoencoder for Convolutional Expression Transfer Using Neural Generative Adversarial Networks](https://mlsanity.com/paper/2112.00532/) (2021)
- [Language-Guided Face Animation by Recurrent StyleGAN-based Generator](https://mlsanity.com/paper/2208.05617/) (2022)
- [AI and 6G into the Metaverse: Fundamentals, Challenges and Future Research Trends](https://mlsanity.com/paper/2208.10921/) (2022)

## Abstract (arXiv)

With the surge in emerging technologies such as Metaverse, spatial computing, and generative AI, the application of facial style transfer has gained a lot of interest from researchers as well as startups enthusiasts alike. StyleGAN methods have paved the way for transfer-learning strategies that could reduce the dependency on the huge volume of data that is available for the training process. However, StyleGAN methods have the tendency of overfitting that results in the introduction of artifacts in the facial images. Studies, such as DualStyleGAN, proposed the use of multipath networks but they require the networks to be trained for a specific style rather than generating a fusion of facial styles at once. In this paper, we propose a FusIon of STyles (FIST) network for facial images that leverages pre-trained multipath style transfer networks to eliminate the problem associated with lack of huge data volume in the training phase along with the fusion of multiple styles at the output. We leverage pre-trained styleGAN networks with an external style pass that use residual modulation block instead of a transform coding block. The method also preserves facial structure, identity, and details via the gated mapping unit introduced in this study. The aforementioned components enable us to train the network with very limited amount of data while generating high-quality stylized images. Our training process adapts curriculum learning strategy to perform efficient, flexible style and model fusion in the generative space. We perform extensive experiments to show the superiority of FISTNet in comparison to existing state-of-the-art methods.