# Image Based Reconstruction of Liquids from 2D Surface Detections

arXiv: https://arxiv.org/abs/2111.11491 · HTML: https://mlsanity.com/paper/2111.11491/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 17 of 17)

- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [SPNets: Differentiable Fluid Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1806.06094/) (2018)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [Visual Closed-Loop Control for Pouring Liquids](https://mlsanity.com/paper/1610.02610/) (2016)
- [Global Transport for Fluid Reconstruction with Learned Self-Supervision](https://mlsanity.com/paper/2104.06031/) (2021)
- [Autonomous Robotic Suction to Clear the Surgical Field for Hemostasis using Image-based Blood Flow Detection](https://mlsanity.com/paper/2010.08441/) (2020)
- [See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content](https://mlsanity.com/paper/1701.02718/) (2017)
- [Perceiving and Reasoning About Liquids Using Fully Convolutional Networks](https://mlsanity.com/paper/1703.01564/) (2017)
- [ScalarFlow: A Large-Scale Volumetric Data Set of Real-world Scalar Transport Flows for Computer Animation and Machine Learning](https://mlsanity.com/paper/2011.10284/) (2020)
- [Model-Predictive Control of Blood Suction for Surgical Hemostasis using Differentiable Fluid Simulations](https://mlsanity.com/paper/2102.01436/) (2021)
- [Can I Pour into It? Robot Imagining Open Containability Affordance of Previously Unseen Objects via Physical Simulations](https://mlsanity.com/paper/2008.02321/) (2020)

## Built on by (most cited first, 4 of 4)

- [A Thermodynamics-informed Active Learning Approach to Perception and Reasoning about Fluids](https://mlsanity.com/paper/2203.05775/) (2022)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [HemoSet: The First Blood Segmentation Dataset for Automation of Hemostasis Management](https://mlsanity.com/paper/2403.16286/) (2024)
- [SplashSplat: Reconstructing Splashing Liquids from Real-World Multi-View Videos](https://mlsanity.com/paper/2609.20818/) (2026)

## Abstract (arXiv)

In this work, we present a solution to the challenging problem of reconstructing liquids from image data. The challenges in reconstructing liquids, which is not faced in previous reconstruction works on rigid and deforming surfaces, lies in the inability to use depth sensing and color features due the variable index of refraction, opacity, and environmental reflections. Therefore, we limit ourselves to only surface detections (i.e. binary mask) of liquids as observations and do not assume any prior knowledge on the liquids properties. A novel optimization problem is posed which reconstructs the liquid as particles by minimizing the error between a rendered surface from the particles and the surface detections while satisfying liquid constraints. Our solvers to this optimization problem are presented and no training data is required to apply them. We also propose a dynamic prediction to seed the reconstruction optimization from the previous time-step. We test our proposed methods in simulation and on two new liquid datasets which we open source so the broader research community can continue developing in this under explored area.