# Self-Supervised Object Detection via Generative Image Synthesis

arXiv: https://arxiv.org/abs/2110.09848 · HTML: https://mlsanity.com/paper/2110.09848/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 60 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 30 of 46)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images](https://mlsanity.com/paper/2002.08988/) (2020)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [A Large-Scale Car Dataset for Fine-Grained Categorization and Verification](https://mlsanity.com/paper/1506.08959/) (2015)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [PCL: Proposal Cluster Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1807.03342/) (2018)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection](https://mlsanity.com/paper/2004.04725/) (2020)

## Built on by (most cited first, 2 of 2)

- [Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Field](https://mlsanity.com/paper/2304.03526/) (2023)
- [Deep Generative Networks for Heterogeneous Augmentation of Cranial Defects](https://mlsanity.com/paper/2308.04883/) (2023)

## Abstract (arXiv)

We present SSOD, the first end-to-end analysis-by synthesis framework with controllable GANs for the task of self-supervised object detection. We use collections of real world images without bounding box annotations to learn to synthesize and detect objects. We leverage controllable GANs to synthesize images with pre-defined object properties and use them to train object detectors. We propose a tight end-to-end coupling of the synthesis and detection networks to optimally train our system. Finally, we also propose a method to optimally adapt SSOD to an intended target data without requiring labels for it. For the task of car detection, on the challenging KITTI and Cityscapes datasets, we show that SSOD outperforms the prior state-of-the-art purely image-based self-supervised object detection method Wetectron. Even without requiring any 3D CAD assets, it also surpasses the state-of-the-art rendering based method Meta-Sim2. Our work advances the field of self-supervised object detection by introducing a successful new paradigm of using controllable GAN-based image synthesis for it and by significantly improving the baseline accuracy of the task. We open-source our code at https://github.com/NVlabs/SSOD.