# Self-supervised Transparent Liquid Segmentation for Robotic Pouring

arXiv: https://arxiv.org/abs/2203.01538 · HTML: https://mlsanity.com/paper/2203.01538/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [Modeling Visual Context is Key to Augmenting Object Detection Datasets](https://mlsanity.com/paper/1807.07428/) (2018)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [Augmentation for small object detection](https://mlsanity.com/paper/1902.07296/) (2019)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [TOM-Net: Learning Transparent Object Matting from a Single Image](https://mlsanity.com/paper/1803.04636/) (2018)
- [TransCut: Transparent Object Segmentation from a Light-Field Image](https://mlsanity.com/paper/1511.06853/) (2015)
- [Visual Closed-Loop Control for Pouring Liquids](https://mlsanity.com/paper/1610.02610/) (2016)
- [Inferring the Material Properties of Granular Media for Robotic Tasks](https://mlsanity.com/paper/2003.08032/) (2020)
- [On the Importance of Visual Context for Data Augmentation in Scene Understanding](https://mlsanity.com/paper/1809.02492/) (2018)
- [Multi-modal Transfer Learning for Grasping Transparent and Specular Objects](https://mlsanity.com/paper/2006.00028/) (2020)
- [Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring](https://mlsanity.com/paper/1903.00650/) (2019)
- [Accurate Pouring with an Autonomous Robot Using an RGB-D Camera](https://mlsanity.com/paper/1810.03303/) (2018)
- [Reasoning About Liquids via Closed-Loop Simulation](https://mlsanity.com/paper/1703.01656/) (2017)
- [Towards Learning to Perceive and Reason About Liquids](https://mlsanity.com/paper/1608.00887/) (2016)

## Built on by (most cited first, 10 of 10)

- [FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation](https://mlsanity.com/paper/2303.02346/) (2023)
- [Physics-Informed Computer Vision: A Review and Perspectives](https://mlsanity.com/paper/2305.18035/) (2023)
- [Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery](https://mlsanity.com/paper/2406.06092/) (2024)
- [The Sound of Water: Inferring Physical Properties from Pouring Liquids](https://mlsanity.com/paper/2411.11222/) (2024)
- [CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image](https://mlsanity.com/paper/2504.11230/) (2025)
- [GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2309.09051/) (2023)
- [You Only Estimate Once: Unified, One-stage, Real-Time Category-level Articulated Object 6D Pose Estimation for Robotic Grasping](https://mlsanity.com/paper/2506.05719/) (2025)
- [Fluid Viscosity Prediction Leveraging Computer Vision and Robot Interaction](https://mlsanity.com/paper/2308.02715/) (2023)
- [SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy](https://mlsanity.com/paper/2510.11566/) (2025)
- [SplashSplat: Reconstructing Splashing Liquids from Real-World Multi-View Videos](https://mlsanity.com/paper/2609.20818/) (2026)

## Abstract (arXiv)

Liquid state estimation is important for robotics tasks such as pouring; however, estimating the state of transparent liquids is a challenging problem. We propose a novel segmentation pipeline that can segment transparent liquids such as water from a static, RGB image without requiring any manual annotations or heating of the liquid for training. Instead, we use a generative model that is capable of translating images of colored liquids into synthetically generated transparent liquid images, trained only on an unpaired dataset of colored and transparent liquid images. Segmentation labels of colored liquids are obtained automatically using background subtraction. Our experiments show that we are able to accurately predict a segmentation mask for transparent liquids without requiring any manual annotations. We demonstrate the utility of transparent liquid segmentation in a robotic pouring task that controls pouring by perceiving the liquid height in a transparent cup. Accompanying video and supplementary materials can be found