# Semi-supervised Cycle-GAN for face photo-sketch translation in the wild

arXiv: https://arxiv.org/abs/2307.10281 · HTML: https://mlsanity.com/paper/2307.10281/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis](https://mlsanity.com/paper/1601.04589/) (2016)
- [CycleGAN, a Master of Steganography](https://mlsanity.com/paper/1712.02950/) (2017)
- [Identity-Aware CycleGAN for Face Photo-Sketch Synthesis and Recognition](https://mlsanity.com/paper/2103.16019/) (2021)
- [Adversarial Self-Defense for Cycle-Consistent GANs](https://mlsanity.com/paper/1908.01517/) (2019)
- [End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning](https://mlsanity.com/paper/1501.07180/) (2015)
- [High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks](https://mlsanity.com/paper/1710.10182/) (2017)
- [Semi-Supervised Learning for Face Sketch Synthesis in the Wild](https://mlsanity.com/paper/1812.04929/) (2018)
- [Multi-Density Sketch-to-Image Translation Network](https://mlsanity.com/paper/2006.10649/) (2020)
- [Face Sketch Synthesis with Style Transfer using Pyramid Column Feature](https://mlsanity.com/paper/2009.08679/) (2020)
- [Unconstrained Face Sketch Synthesis via Perception-Adaptive Network and A New Benchmark](https://mlsanity.com/paper/2112.01019/) (2021)
- [Random Sampling for Fast Face Sketch Synthesis](https://mlsanity.com/paper/1701.01911/) (2017)
- [Content-Adaptive Sketch Portrait Generation by Decompositional Representation Learning](https://mlsanity.com/paper/1710.01453/) (2017)

## Built on by (most cited first, 2 of 2)

- [CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models](https://mlsanity.com/paper/2408.01233/) (2024)
- [Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization](https://mlsanity.com/paper/2410.02628/) (2024)

## Abstract (arXiv)

The performance of face photo-sketch translation has improved a lot thanks to deep neural networks. GAN based methods trained on paired images can produce high-quality results under laboratory settings. Such paired datasets are, however, often very small and lack diversity. Meanwhile, Cycle-GANs trained with unpaired photo-sketch datasets suffer from the steganography phenomenon, which makes them not effective to face photos in the wild. In this paper, we introduce a semi-supervised approach with a noise-injection strategy, named Semi-Cycle-GAN (SCG), to tackle these problems. For the first problem, we propose a em pseudo sketch feature representation for each input photo composed from a small reference set of photo-sketch pairs, and use the resulting em pseudo pairs to supervise a photo-to-sketch generator Gp2s. The outputs of Gp2s can in turn help to train a sketch-to-photo generator Gs2p in a self-supervised manner. This allows us to train Gp2s and Gs2p using a small reference set of photo-sketch pairs together with a large face photo dataset (without ground-truth sketches). For the second problem, we show that the simple noise-injection strategy works well to alleviate the steganography effect in SCG and helps to produce more reasonable sketch-to-photo results with less overfitting than fully supervised approaches. Experiments show that SCG achieves competitive performance on public benchmarks and superior results on photos in the wild.