# SuperCaustics: Real-time, open-source simulation of transparent objects for deep learning applications

arXiv: https://arxiv.org/abs/2107.11008 · HTML: https://mlsanity.com/paper/2107.11008/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 23 Jul 2021.

## Builds on (most cited first, 7 of 7)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [Soccer on Your Tabletop](https://mlsanity.com/paper/1806.00890/) (2018)
- [AI Playground: Unreal Engine-based Data Ablation Tool for Deep Learning](https://mlsanity.com/paper/2007.06153/) (2020)

## Built on by (most cited first, 5 of 5)

- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [Fully Automated Tree Topology Estimation and Artery-Vein Classification](https://mlsanity.com/paper/2202.02382/) (2022)
- [Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks](https://mlsanity.com/paper/2503.04308/) (2025)
- [N-SfC: Robust and Fast Shape Estimation from Caustic Images](https://mlsanity.com/paper/2112.06705/) (2021)
- [Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly](https://mlsanity.com/paper/2603.20290/) (2026)

## Abstract (arXiv)

Transparent objects are a very challenging problem in computer vision. They are hard to segment or classify due to their lack of precise boundaries, and there is limited data available for training deep neural networks. As such, current solutions for this problem employ rigid synthetic datasets, which lack flexibility and lead to severe performance degradation when deployed on real-world scenarios. In particular, these synthetic datasets omit features such as refraction, dispersion and caustics due to limitations in the rendering pipeline. To address this issue, we present SuperCaustics, a real-time, open-source simulation of transparent objects designed for deep learning applications. SuperCaustics features extensive modules for stochastic environment creation; uses hardware ray-tracing to support caustics, dispersion, and refraction; and enables generating massive datasets with multi-modal, pixel-perfect ground truth annotations. To validate our proposed system, we trained a deep neural network from scratch to segment transparent objects in difficult lighting scenarios. Our neural network achieved performance comparable to the state-of-the-art on a real-world dataset using only 10% of the training data and in a fraction of the training time. Further experiments show that a model trained with SuperCaustics can segment different types of caustics, even in images with multiple overlapping transparent objects. To the best of our knowledge, this is the first such result for a model trained on synthetic data. Both our open-source code and experimental data are freely available online.