# On-the-fly Object Detection using StyleGAN with CLIP Guidance

arXiv: https://arxiv.org/abs/2210.16742 · HTML: https://mlsanity.com/paper/2210.16742/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 30 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [On the Units of GANs (Extended Abstract)](https://mlsanity.com/paper/1901.09887/) (2019)
- [An Interactive Interpretability System for Breast Cancer Screening with Deep Learning](https://mlsanity.com/paper/2210.08979/) (2022)

## Abstract (arXiv)

We present a fully automated framework for building object detectors on satellite imagery without requiring any human annotation or intervention. We achieve this by leveraging the combined power of modern generative models (e.g., StyleGAN) and recent advances in multi-modal learning (e.g., CLIP). While deep generative models effectively encode the key semantics pertinent to a data distribution, this information is not immediately accessible for downstream tasks, such as object detection. In this work, we exploit CLIP's ability to associate image features with text descriptions to identify neurons in the generator network, which are subsequently used to build detectors on-the-fly.