# Controlling Style and Semantics in Weakly-Supervised Image Generation

arXiv: https://arxiv.org/abs/1912.03161 · HTML: https://mlsanity.com/paper/1912.03161/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2019.

## Built on by (most cited first, 11 of 11)

- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Learning Generative Models of Textured 3D Meshes from Real-World Images](https://mlsanity.com/paper/2103.15627/) (2021)
- [SceneComposer: Any-Level Semantic Image Synthesis](https://mlsanity.com/paper/2211.11742/) (2022)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style](https://mlsanity.com/paper/2103.13722/) (2021)
- [Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](https://mlsanity.com/paper/2110.02891/) (2021)
- [DT2I: Dense Text-to-Image Generation from Region Descriptions](https://mlsanity.com/paper/2204.02035/) (2022)
- [Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects](https://mlsanity.com/paper/2410.01816/) (2024)

## Abstract (arXiv)

We propose a weakly-supervised approach for conditional image generation of complex scenes where a user has fine control over objects appearing in the scene. We exploit sparse semantic maps to control object shapes and classes, as well as textual descriptions or attributes to control both local and global style. In order to condition our model on textual descriptions, we introduce a semantic attention module whose computational cost is independent of the image resolution. To further augment the controllability of the scene, we propose a two-step generation scheme that decomposes background and foreground. The label maps used to train our model are produced by a large-vocabulary object detector, which enables access to unlabeled data and provides structured instance information. In such a setting, we report better FID scores compared to fully-supervised settings where the model is trained on ground-truth semantic maps. We also showcase the ability of our model to manipulate a scene on complex datasets such as COCO and Visual Genome.