# Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics

arXiv: https://arxiv.org/abs/2005.05659 · HTML: https://mlsanity.com/paper/2005.05659/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 11 of 11)

- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Cross-Modality Time-Variant Relation Learning for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2305.08522/) (2023)
- [Sim2Real Instance-Level Style Transfer for 6D Pose Estimation](https://mlsanity.com/paper/2203.02069/) (2022)
- [Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translation](https://mlsanity.com/paper/2203.09454/) (2022)
- [Informed Pre-Training on Prior Knowledge](https://mlsanity.com/paper/2205.11433/) (2022)
- [External Camera-based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors](https://mlsanity.com/paper/2303.03797/) (2023)
- [SynPick: A Dataset for Dynamic Bin Picking Scene Understanding](https://mlsanity.com/paper/2107.04852/) (2021)
- [Object-level 3D Semantic Mapping using a Network of Smart Edge Sensors](https://mlsanity.com/paper/2211.11354/) (2022)
- [Predicting Physical Object Properties from Video](https://mlsanity.com/paper/2206.00930/) (2022)
- [Learning from SAM: Harnessing a Foundation Model for Sim2Real Adaptation by Regularization](https://mlsanity.com/paper/2309.15562/) (2023)
- [Efficient Human-Contact Representation for Human-Scene Interaction](https://mlsanity.com/paper/2608.09388/) (2026)

## Abstract (arXiv)

Training data is the key ingredient for deep learning approaches, but difficult to obtain for the specialized domains often encountered in robotics. We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks such as semantic segmentation, object detection, and correspondence or pose estimation. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. The arranged scenes are rendered using high-quality rasterization with randomized appearance and material parameters. Noise and other transformations introduced by the camera sensors are simulated. Our pipeline can be run online during training of a deep neural network, yielding applications in life-long learning and in iterative render-and-compare approaches. We demonstrate the usability by learning semantic segmentation on the challenging YCB-Video dataset without actually using any training frames, where our method achieves performance comparable to a conventionally trained model. Additionally, we show successful application in a real-world regrasping system.