# Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision

arXiv: https://arxiv.org/abs/2302.14470 · HTML: https://mlsanity.com/paper/2302.14470/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [The relativistic discriminator: a key element missing from standard GAN](https://mlsanity.com/paper/1807.00734/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [Escaping Plato's Cave: 3D Shape From Adversarial Rendering](https://mlsanity.com/paper/1811.11606/) (2018)
- [SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization](https://mlsanity.com/paper/1912.07109/) (2019)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [Deep Fluids: A Generative Network for Parameterized Fluid Simulations](https://mlsanity.com/paper/1806.02071/) (2018)
- [V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/1711.07399/) (2017)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation](https://mlsanity.com/paper/1808.01838/) (2018)
- [Self-Supervised Monocular Scene Flow Estimation](https://mlsanity.com/paper/2004.04143/) (2020)
- [SPNets: Differentiable Fluid Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1806.06094/) (2018)
- [Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data](https://mlsanity.com/paper/2206.06577/) (2022)
- [UPFlow: Upsampling Pyramid for Unsupervised Optical Flow Learning](https://mlsanity.com/paper/2012.00212/) (2020)
- [Self-Supervised Multi-Frame Monocular Scene Flow](https://mlsanity.com/paper/2105.02216/) (2021)
- [Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes](https://mlsanity.com/paper/1908.06316/) (2019)
- [PWOC-3D: Deep Occlusion-Aware End-to-End Scene Flow Estimation](https://mlsanity.com/paper/1904.06116/) (2019)
- [Global Transport for Fluid Reconstruction with Learned Self-Supervision](https://mlsanity.com/paper/2104.06031/) (2021)
- [Unsupervised convolutional neural networks for motion estimation](https://mlsanity.com/paper/1601.06087/) (2016)
- [Learning Residual Flow as Dynamic Motion from Stereo Videos](https://mlsanity.com/paper/1909.06999/) (2019)
- [Lagrangian Neural Style Transfer for Fluids](https://mlsanity.com/paper/2005.00803/) (2020)
- [Transport-Based Neural Style Transfer for Smoke Simulations](https://mlsanity.com/paper/1905.07442/) (2019)

## Built on by (most cited first, 5 of 5)

- [TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videos](https://mlsanity.com/paper/2508.09811/) (2025)
- [HelmFluid: Learning Helmholtz Dynamics for Interpretable Fluid Prediction](https://mlsanity.com/paper/2310.10565/) (2023)
- [Unraveling Normal Anatomy via Fluid-Driven Anomaly Randomization](https://mlsanity.com/paper/2501.13370/) (2025)
- [Learning an Implicit Physics Model for Image-based Fluid Simulation](https://mlsanity.com/paper/2508.08254/) (2025)
- [SmokeSVD: Smoke Reconstruction from A Single View via Progressive Novel View Synthesis and Refinement with Diffusion Models](https://mlsanity.com/paper/2507.12156/) (2025)

## Abstract (arXiv)

We address the challenging problem of jointly inferring the 3D flow and volumetric densities moving in a fluid from a monocular input video with a deep neural network. Despite the complexity of this task, we show that it is possible to train the corresponding networks without requiring any 3D ground truth for training. In the absence of ground truth data we can train our model with observations from real-world capture setups instead of relying on synthetic reconstructions. We make this unsupervised training approach possible by first generating an initial prototype volume which is then moved and transported over time without the need for volumetric supervision. Our approach relies purely on image-based losses, an adversarial discriminator network, and regularization. Our method can estimate long-term sequences in a stable manner, while achieving closely matching targets for inputs such as rising smoke plumes.