mlsanityData to 7 Oct 2026

DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception

arXiv 2403.13304 ↗First posted 20 Mar 2024cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Current perceptive models heavily depend on resource-intensive datasets, prompting the need for innovative solutions. Leveraging recent advances in diffusion models, synthetic data, by constructing image inputs from various annotations, proves beneficial for downstream tasks. While prior methods have separately addressed generative and perceptive models, DetDiffusion, for the first time, harmonizes both, tackling the challenges in generating effective data for perceptive models. To enhance image generation with perceptive models, we introduce perception-aware loss (P.A. loss) through segmentation, improving both quality and controllability. To boost the performance of specific perceptive models, our method customizes data augmentation by extracting and utilizing perception-aware attribute (P.A. Attr) during generation. Experimental results from the object detection task highlight DetDiffusion's superior performance, establishing a new state-of-the-art in layout-guided generation. Furthermore, image syntheses from DetDiffusion can effectively augment training data, significantly enhancing downstream detection performance.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

48 of 57 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2023Segment Anythingcited by 8,158
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
36 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  3. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  6. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  7. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  8. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  9. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  10. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  11. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  12. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 258
  13. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  14. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  15. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  16. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  17. 2022Semantic Image Synthesis via Diffusion Modelscited by 131
  18. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  19. 2018GAN Augmentation: Augmenting Training Data using Generative Adversarial Networkscited by 107
  20. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  21. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  22. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  23. 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
  24. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesiscited by 75
  25. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generationcited by 65
  26. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  27. 2021MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Drivingcited by 41
  28. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  29. 2020Deflating Dataset Bias Using Synthetic Data Augmentationcited by 37
  30. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  31. 2021High-Resolution Complex Scene Synthesis with Transformerscited by 31
  32. 2023DiffGuard: Semantic Mismatch-Guided Out-of-Distribution Detection using Pre-trained Diffusion Modelscited by 25
  33. 2022Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routingcited by 18
  34. 2024Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Expertscited by 16
  35. 2022Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translationcited by 9
  36. 2023Domain Adaptation of Synthetic Driving Datasets for Real-World Autonomous Drivingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 15; 2026: 3.

32 papers.

Sort:

By area:

  1. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  2. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  3. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  4. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  5. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  6. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  7. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  8. 2024Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentationcited by 12
  9. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  10. 2024OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generationcited by 10
  11. 2025DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Modelscited by 10
  12. 2025Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Methodcited by 8
20 more
  1. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Modelscited by 7
  2. 2025ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detectioncited by 6
  3. 2025A Unified Image-Dense Annotation Generation Model for Underwater Scenescited by 5
  4. 2025ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Drivingcited by 5
  5. 2025Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesiscited by 4
  6. 2024Training-Free Point Cloud Recognition Based on Geometric and Semantic Information Fusioncited by 2
  7. 2025Gaussian Splatting is an Effective Data Generator for 3D Object Detectioncited by 2
  8. 2025Boosting Domain Generalized and Adaptive Detection with Diffusion Models: Fitness, Generalization, and Transferabilitycited by 2
  9. 2024LocRef-Diffusion:Tuning-Free Layout and Appearance-Guided Generationcited by 1
  10. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  11. 2025Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detectioncited by 1
  12. 2025ViCTr: Vital Consistency Transfer for Pathology Aware Image Synthesiscited by 1
  13. 2025MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generationcited by 1
  14. 2025Subtyping Breast Lesions via Generative Augmentation based Long-tailed Recognition in Ultrasoundcited by 1
  15. 2024BDC-Occ: Binarized Deep Convolution Unit For Binarized Occupancy Networkcited by 0
  16. 2025RGBX-DiffusionDet: A Framework for Multi-Modal RGB-X Object Detection Using DiffusionDetcited by 0
  17. 2025Taming Generative Synthetic Data for X-ray Prohibited Item Detectioncited by 0
  18. 2026Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexitycited by 0
  19. 2026Visual Prototype Conditioned Focal Region Generation for UAV-Based Object Detectioncited by 0
  20. 2026RailSyn: Diagnosis-Guided Image Generation for Traceable Data Completion in Railway Foreign Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models
  2. 2026MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
  3. 2024FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models
  4. 2023GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation
  5. 2024Scaling Properties of Diffusion Models for Perceptual Tasks
  6. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspect
  7. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
  8. 2023DDP: Diffusion Model for Dense Visual Prediction