# InstaGen: Enhancing Object Detection by Training on Synthetic Dataset

arXiv: https://arxiv.org/abs/2402.05937 · HTML: https://mlsanity.com/paper/2402.05937/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 38 of its 43 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [TOOD: Task-aligned One-stage Object Detection](https://mlsanity.com/paper/2108.07755/) (2021)
- [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022)
- [PromptDet: Towards Open-vocabulary Detection using Uncurated Images](https://mlsanity.com/paper/2203.16513/) (2022)
- [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023)
- [Probabilistic two-stage detection](https://mlsanity.com/paper/2103.07461/) (2021)
- [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023)
- [Exploiting Unlabeled Data with Vision and Language Models for Object Detection](https://mlsanity.com/paper/2207.08954/) (2022)

## Built on by (most cited first, 26 of 26)

- [ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models](https://mlsanity.com/paper/2405.15199/) (2024)
- [DODA: Adapting Object Detectors to Dynamic Agricultural Environments in Real-Time with Diffusion](https://mlsanity.com/paper/2403.18334/) (2024)
- [RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2405.19854/) (2024)
- [Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification](https://mlsanity.com/paper/2409.17091/) (2024)
- [Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs](https://mlsanity.com/paper/2503.12999/) (2025)
- [Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data](https://mlsanity.com/paper/2505.10551/) (2025)
- [Advancing Visual Large Language Model for Multi-granular Versatile Perception](https://mlsanity.com/paper/2507.16213/) (2025)
- [No Annotations for Object Detection in Art through Stable Diffusion](https://mlsanity.com/paper/2412.06286/) (2024)
- [The Role of Video Generation in Enhancing Data-Limited Action Understanding](https://mlsanity.com/paper/2505.19495/) (2025)
- [MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2510.14792/) (2025)
- [On the Difficulty of Learning a Meta-network for Training Data Selection](https://mlsanity.com/paper/2606.00571/) (2026)
- [Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection](https://mlsanity.com/paper/2408.02891/) (2024)
- [Boosting Few-Shot Detection with Large Language Models and Layout-to-Image Synthesis](https://mlsanity.com/paper/2410.06841/) (2024)
- [Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding](https://mlsanity.com/paper/2412.00684/) (2024)
- [Understanding Trade offs When Conditioning Synthetic Data](https://mlsanity.com/paper/2507.02217/) (2025)
- [The Impact of Synthetic Data on Object Detection Model Performance: A Comparative Analysis with Real-World Data](https://mlsanity.com/paper/2510.12208/) (2025)
- [Generative Spatiotemporal Data Augmentation](https://mlsanity.com/paper/2512.12508/) (2025)
- [OCDet: Object Center Detection via Bounding Box-Aware Heatmap Prediction on Edge Devices with NPUs](https://mlsanity.com/paper/2411.15653/) (2024)
- [Crucial-Diff: A Unified Diffusion Model for Crucial Image and Annotation Synthesis in Data-scarce Scenarios](https://mlsanity.com/paper/2507.09915/) (2025)
- [High-Quality Proposal Encoding and Cascade Denoising for Imaginary Supervised Object Detection](https://mlsanity.com/paper/2511.08018/) (2025)
- [Object-Centric Data Synthesis for Category-level Object Detection](https://mlsanity.com/paper/2511.23450/) (2025)
- [Auto-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2512.16077/) (2025)
- [A Sanity Check on Composed Image Retrieval](https://mlsanity.com/paper/2604.12904/) (2026)
- [Beyond Visual Cues: Semantic-Driven Token Filtering and Expert Routing for Anytime Person ReID](https://mlsanity.com/paper/2604.15090/) (2026)
- [Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance](https://mlsanity.com/paper/2606.19817/) (2026)
- [UniScale: Arbitrary-Scale Industrial Anomaly Generation](https://mlsanity.com/paper/2608.07864/) (2026)

## Abstract (arXiv)

In this paper, we present a novel paradigm to enhance the ability of object detector, e.g., expanding categories or improving detection performance, by training on synthetic dataset generated from diffusion models. Specifically, we integrate an instance-level grounding head into a pre-trained, generative diffusion model, to augment it with the ability of localising instances in the generated images. The grounding head is trained to align the text embedding of category names with the regional visual feature of the diffusion model, using supervision from an off-the-shelf object detector, and a novel self-training scheme on (novel) categories not covered by the detector. We conduct thorough experiments to show that, this enhanced version of diffusion model, termed as InstaGen, can serve as a data synthesizer, to enhance object detectors by training on its generated samples, demonstrating superior performance over existing state-of-the-art methods in open-vocabulary (+4.5 AP) and data-sparse (+1.2 to 5.2 AP) scenarios. Project page with code: https://fcjian.github.io/InstaGen.