# GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation

arXiv: https://arxiv.org/abs/2306.04607 · HTML: https://mlsanity.com/paper/2306.04607/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 44 of its 64 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://mlsanity.com/paper/2212.06909/) (2022)

## Built on by (most cited first, 30 of 56)

- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024)
- [MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes](https://mlsanity.com/paper/2405.14475/) (2024)
- [DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control](https://mlsanity.com/paper/2312.03048/) (2023)
- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.07132/) (2024)
- [AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generation](https://mlsanity.com/paper/2411.15497/) (2024)
- [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://mlsanity.com/paper/2401.08045/) (2024)
- [ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models](https://mlsanity.com/paper/2405.15199/) (2024)
- [Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2506.05872/) (2025)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment](https://mlsanity.com/paper/2405.00557/) (2024)
- [DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Models](https://mlsanity.com/paper/2502.14779/) (2025)
- [Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis](https://mlsanity.com/paper/2311.18435/) (2023)
- [CC-Diff: Enhancing Contextual Coherence in Remote Sensing Image Synthesis](https://mlsanity.com/paper/2412.08464/) (2024)
- [QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots](https://mlsanity.com/paper/2508.02512/) (2025)
- [Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models](https://mlsanity.com/paper/2410.23114/) (2024)
- [ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detection](https://mlsanity.com/paper/2510.15783/) (2025)
- [ZoDi: Zero-Shot Domain Adaptation with Diffusion-Based Image Transfer](https://mlsanity.com/paper/2403.13652/) (2024)
- [STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation](https://mlsanity.com/paper/2506.13138/) (2025)
- [Object Fidelity Diffusion for Remote Sensing Image Generation](https://mlsanity.com/paper/2508.10801/) (2025)
- [SIGMA-GEN: Structure and Identity Guided Multi-subject Assembly for Image Generation](https://mlsanity.com/paper/2510.06469/) (2025)
- [Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language](https://mlsanity.com/paper/2406.20085/) (2024)
- [Training-Free Layout-to-Image Generation with Marginal Attention Constraints](https://mlsanity.com/paper/2411.10495/) (2024)
- [Detail++: Training-Free Detail Enhancer for T2I Diffusion Models](https://mlsanity.com/paper/2507.17853/) (2025)
- [FICGen: Frequency-Inspired Contextual Disentanglement for Layout-driven Degraded Image Generation](https://mlsanity.com/paper/2509.01107/) (2025)
- [Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis](https://mlsanity.com/paper/2510.18229/) (2025)
- [Gaussian Splatting is an Effective Data Generator for 3D Object Detection](https://mlsanity.com/paper/2504.16740/) (2025)
- [DIVE: Inverting Conditional Diffusion Models for Discriminative Tasks](https://mlsanity.com/paper/2504.17253/) (2025)
- [Boosting Domain Generalized and Adaptive Detection with Diffusion Models: Fitness, Generalization, and Transferability](https://mlsanity.com/paper/2506.21042/) (2025)

## Abstract (arXiv)

Diffusion models have attracted significant attention due to the remarkable ability to create content and generate data for tasks like image classification. However, the usage of diffusion models to generate the high-quality object detection data remains an underexplored area, where not only image-level perceptual quality but also geometric conditions such as bounding boxes and camera views are essential. Previous studies have utilized either copy-paste synthesis or layout-to-image (L2I) generation with specifically designed modules to encode the semantic layouts. In this paper, we propose the GeoDiffusion, a simple framework that can flexibly translate various geometric conditions into text prompts and empower pre-trained text-to-image (T2I) diffusion models for high-quality detection data generation. Unlike previous L2I methods, our GeoDiffusion is able to encode not only the bounding boxes but also extra geometric conditions such as camera views in self-driving scenes. Extensive experiments demonstrate GeoDiffusion outperforms previous L2I methods while maintaining 4x training time faster. To the best of our knowledge, this is the first work to adopt diffusion models for layout-to-image generation with geometric conditions and demonstrate that L2I-generated images can be beneficial for improving the performance of object detectors.