# DiffusionEngine: Diffusion Model is Scalable Data Engine for Object Detection

arXiv: https://arxiv.org/abs/2309.03893 · HTML: https://mlsanity.com/paper/2309.03893/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 36 of its 44 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptation](https://mlsanity.com/paper/1803.11365/) (2018)

## Built on by (most cited first, 23 of 23)

- [DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control](https://mlsanity.com/paper/2312.03048/) (2023)
- [Interpretable Diffusion via Information Decomposition](https://mlsanity.com/paper/2310.07972/) (2023)
- [ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models](https://mlsanity.com/paper/2405.15199/) (2024)
- [DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Mode](https://mlsanity.com/paper/2503.12838/) (2025)
- [Image Understanding Makes for A Good Tokenizer for Image Generation](https://mlsanity.com/paper/2411.04406/) (2024)
- [Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2506.05872/) (2025)
- [FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models](https://mlsanity.com/paper/2403.20105/) (2024)
- [ByteEdit: Boost, Comply and Accelerate Generative Image Editing](https://mlsanity.com/paper/2404.04860/) (2024)
- [ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding](https://mlsanity.com/paper/2408.16314/) (2024)
- [ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentation](https://mlsanity.com/paper/2311.17121/) (2023)
- [DODA: Adapting Object Detectors to Dynamic Agricultural Environments in Real-Time with Diffusion](https://mlsanity.com/paper/2403.18334/) (2024)
- [DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognition](https://mlsanity.com/paper/2403.05170/) (2024)
- [Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation](https://mlsanity.com/paper/2501.00873/) (2025)
- [Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervision](https://mlsanity.com/paper/2507.20976/) (2025)
- [Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding](https://mlsanity.com/paper/2412.00684/) (2024)
- [Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentation](https://mlsanity.com/paper/2412.08116/) (2024)
- [Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detection](https://mlsanity.com/paper/2504.17076/) (2025)
- [Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns](https://mlsanity.com/paper/2506.00997/) (2025)
- [Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning](https://mlsanity.com/paper/2510.04770/) (2025)
- [FLORA: Efficient Synthetic Data Generation for Object Detection in Low-Data Regimes via finetuning Flux LoRA](https://mlsanity.com/paper/2508.21712/) (2025)
- [High-Quality Proposal Encoding and Cascade Denoising for Imaginary Supervised Object Detection](https://mlsanity.com/paper/2511.08018/) (2025)
- [Object-Centric Data Synthesis for Category-level Object Detection](https://mlsanity.com/paper/2511.23450/) (2025)
- [GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detection](https://mlsanity.com/paper/2601.07273/) (2026)

## Abstract (arXiv)

Data is the cornerstone of deep learning. This paper reveals that the recently developed Diffusion Model is a scalable data engine for object detection. Existing methods for scaling up detection-oriented data often require manual collection or generative models to obtain target images, followed by data augmentation and labeling to produce training pairs, which are costly, complex, or lacking diversity. To address these issues, we presentDiffusionEngine (DE), a data scaling-up engine that provides high-quality detection-oriented training pairs in a single stage. DE consists of a pre-trained diffusion model and an effective Detection-Adapter, contributing to generating scalable, diverse and generalizable detection data in a plug-and-play manner. Detection-Adapter is learned to align the implicit semantic and location knowledge in off-the-shelf diffusion models with detection-aware signals to make better bounding-box predictions. Additionally, we contribute two datasets, i.e., COCO-DE and VOC-DE, to scale up existing detection benchmarks for facilitating follow-up research. Extensive experiments demonstrate that data scaling-up via DE can achieve significant improvements in diverse scenarios, such as various detection algorithms, self-supervised pre-training, data-sparse, label-scarce, cross-domain, and semi-supervised learning. For example, when using DE with a DINO-based adapter to scale up data, mAP is improved by 3.1% on COCO, 7.6% on VOC, and 11.5% on Clipart.