# Diffuse the object, keep its label: curating detector training data from a few unlabeled photographs via VLM-built 3D vegetation scenes

arXiv: https://arxiv.org/abs/2608.09691 · HTML: https://mlsanity.com/paper/2608.09691/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code](https://mlsanity.com/paper/2403.01248/) (2024)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation](https://mlsanity.com/paper/2506.07826/) (2025)
- [Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models](https://mlsanity.com/paper/2606.02580/) (2026)
- [Coarse-to-Real: Generative Rendering for Populated Dynamic Scenes](https://mlsanity.com/paper/2601.22301/) (2026)
- [MineInsight: A Multi-sensor Dataset for Humanitarian Demining Robotics in Off-Road Environments](https://mlsanity.com/paper/2506.04842/) (2025)
- [Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation](https://mlsanity.com/paper/2504.08473/) (2025)
- [Leafy Spurge Dataset: Real-world Weed Classification Within Aerial Drone Imagery](https://mlsanity.com/paper/2405.03702/) (2024)
- [Rendering-Refined Stable Diffusion for Privacy Compliant Synthetic Data](https://mlsanity.com/paper/2412.06248/) (2024)
- [Exploring Model Quantization in GenAI-based Image Inpainting and Detection of Arable Plants](https://mlsanity.com/paper/2503.02420/) (2025)
- [A multi-robot system for the detection of explosive devices](https://mlsanity.com/paper/2404.14167/) (2024)
- [A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation](https://mlsanity.com/paper/2512.08747/) (2025)
- [ForestPersons: A Large-Scale Dataset for Under-Canopy Missing Person Detection](https://mlsanity.com/paper/2603.02541/) (2026)

## Built on by (most cited first, 1 of 1)

- [Domain generalization and synthetic data in object detection: the enabler, the probe, and the gap](https://mlsanity.com/paper/2610.00030/) (2026)

## Abstract (arXiv)

Labeled images of small objects hidden in vegetation are scarce, and detectors trained on them generalize poorly across sites. Rather than reusing labels collected at another site, we synthesize labeled training images from a handful of unlabeled photographs of the deployment site itself. A vision--language model generates a coarse 3D vegetation scene from one photograph; placing 3D object meshes in the scene yields bounding boxes, segmentation masks, and per-instance occlusion directly from the scene geometry, without manual annotation. A lightweight adapter fine-tuned on the photographs conditions a diffusion pass that re-textures the renders, and a graded mask-lock sets how much diffusion may touch the object itself. In our runs this grade was the most influential curation choice: lightly diffusing the object improves minority-class recall over fully protecting its pixels, while unrestricted diffusion dissolves it. Trained on these images, a standard detector matched or exceeded its counterpart trained on a larger labeled dataset of real images from a different site, consistently across seeds on a humanitarian-demining benchmark; the comparison is thus unsupervised site adaptation from a handful of photographs against conventional cross-site label reuse. In our ablations the gains were largely insensitive to the photograph and crop budgets, and in-domain accuracy did not predict cross-site performance.