Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI dataset
Not on the map. Why?
Abstract (arXiv)
We present TransProteus, a dataset, and methods for predicting the 3D structure, masks, and properties of materials, liquids, and objects inside transparent vessels from a single image without prior knowledge of the image source and camera parameters. Manipulating materials in transparent containers is essential in many fields and depends heavily on vision. This work supplies a new procedurally generated dataset consisting of 50k images of liquids and solid objects inside transparent containers. The image annotations include 3D models, material properties (color/transparency/roughness...), and segmentation masks for the vessel and its content. The synthetic (CGI) part of the dataset was procedurally generated using 13k different objects, 500 different environments (HDRI), and 1450 material textures (PBR) combined with simulated liquids and procedurally generated vessels. In addition, we supply 104 real-world images of objects inside transparent vessels with depth maps of both the vessel and its content. We propose a camera agnostic method that predicts 3D models from an image as an XYZ map. This allows the trained net to predict the 3D model as a map with XYZ coordinates per pixel without prior knowledge of the image source. To calculate the training loss, we use the distance between pairs of points inside the 3D model instead of the absolute XYZ coordinates. This makes the loss function translation invariant. We use this to predict 3D models of vessels and their content from a single image. Finally, we demonstrate a net that uses a single image to predict the material properties of the vessel content and surface.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 15 Sep 2021.
Builds on
28 of 56 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
- 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
- 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
- 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
- 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
- 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
- 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
- 2016Single-Image Depth Perception in the Wildcited by 179
- 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
16 more
- 2019ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulationcited by 124
- 2020Segmenting Transparent Objects in the Wildcited by 98
- 2017Intel RealSense Stereoscopic Depth Camerascited by 82
- 2020DiverseDepth: Affine-invariant Depth Prediction Using Diverse Datacited by 60
- 2018TOM-Net: Learning Transparent Object Matting from a Single Imagecited by 34
- 2020Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapescited by 32
- 2021Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real Worldcited by 30
- 2018Learning Single-Image Depth from Videos using Quality Assessment Networkscited by 25
- 2017See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Contentcited by 11
- 2017Perceiving and Reasoning About Liquids Using Fully Convolutional Networkscited by 10
- 2020Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applicationscited by 6
- 2017Training and Testing Object Detectors with Virtual Imagescited by 5
- 2021Computer vision for liquid samples in hospitals and medical labs using hierarchical image segmentation and relations predictioncited by 5
- 2014Computer vision-based recognition of liquid surfaces and phase boundaries in transparent vessels, with emphasis on chemistry applicationscited by 4
- 2016Tracing liquid level and material boundaries in transparent vessels using the graph cut computer vision approachcited by 3
- 2017Setting an attention region for convolutional neural networks using region selective features, for recognition of materials within glass vesselscited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.