# Evaluating Text-to-Image Diffusion Models for Texturing Synthetic Data

arXiv: https://arxiv.org/abs/2411.10164 · HTML: https://mlsanity.com/paper/2411.10164/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 6 of its 41 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 6 of 6)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [Medical diffusion on a budget: Textual Inversion for medical image generation](https://mlsanity.com/paper/2303.13430/) (2023)
- [The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better](https://mlsanity.com/paper/2406.05184/) (2024)
- [Generating Images with 3D Annotations Using Diffusion Models](https://mlsanity.com/paper/2306.08103/) (2023)

## Built on by (most cited first, 1 of 1)

- [Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis](https://mlsanity.com/paper/2508.05580/) (2025)

## Abstract (arXiv)

Building generic robotic manipulation systems often requires large amounts of real-world data, which can be dificult to collect. Synthetic data generation offers a promising alternative, but limiting the sim-to-real gap requires significant engineering efforts. To reduce this engineering effort, we investigate the use of pretrained text-to-image diffusion models for texturing synthetic images and compare this approach with using random textures, a common domain randomization technique in synthetic data generation. We focus on generating object-centric representations, such as keypoints and segmentation masks, which are important for robotic manipulation and require precise annotations. We evaluate the efficacy of the texturing methods by training models on the synthetic data and measuring their performance on real-world datasets for three object categories: shoes, T-shirts, and mugs. Surprisingly, we find that texturing using a diffusion model performs on par with random textures, despite generating seemingly more realistic images. Our results suggest that, for now, using diffusion models for texturing does not benefit synthetic data generation for robotics. The code, data and trained models are available at urlhttps://github.com/tlpss/diffusing-synthetic-data.git.