# CIMGEN: Controlled Image Manipulation by Finetuning Pretrained Generative Models on Limited Data

arXiv: https://arxiv.org/abs/2401.13006 · HTML: https://mlsanity.com/paper/2401.13006/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 13 of 13)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [GIRAFFE HD: A High-Resolution 3D-aware Generative Model](https://mlsanity.com/paper/2203.14954/) (2022)
- [PD-GAN: Probabilistic Diverse GAN for Image Inpainting](https://mlsanity.com/paper/2105.02201/) (2021)
- [InsetGAN for Full-Body Image Generation](https://mlsanity.com/paper/2203.07293/) (2022)
- [Collaging Class-specific GANs for Semantic Image Synthesis](https://mlsanity.com/paper/2110.04281/) (2021)
- [edge-SR: Super-Resolution For The Masses](https://mlsanity.com/paper/2108.10335/) (2021)
- [Adversarially Optimized Mixup for Robust Classification](https://mlsanity.com/paper/2103.11589/) (2021)

## Abstract (arXiv)

Content creation and image editing can benefit from flexible user controls. A common intermediate representation for conditional image generation is a semantic map, that has information of objects present in the image. When compared to raw RGB pixels, the modification of semantic map is much easier. One can take a semantic map and easily modify the map to selectively insert, remove, or replace objects in the map. The method proposed in this paper takes in the modified semantic map and alter the original image in accordance to the modified map. The method leverages traditional pre-trained image-to-image translation GANs, such as CycleGAN or Pix2Pix GAN, that are fine-tuned on a limited dataset of reference images associated with the semantic maps. We discuss the qualitative and quantitative performance of our technique to illustrate its capacity and possible applications in the fields of image forgery and image editing. We also demonstrate the effectiveness of the proposed image forgery technique in thwarting the numerous deep learning-based image forensic techniques, highlighting the urgent need to develop robust and generalizable image forensic tools in the fight against the spread of fake media.