# Volumetric Cloud Field Reconstruction

arXiv: https://arxiv.org/abs/2311.17657 · HTML: https://mlsanity.com/paper/2311.17657/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion](https://mlsanity.com/paper/2302.10109/) (2023)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [EfficientNeRF: Efficient Neural Radiance Fields](https://mlsanity.com/paper/2206.00878/) (2022)
- [Vision Transformer for NeRF-Based View Synthesis from a Single Input Image](https://mlsanity.com/paper/2207.05736/) (2022)
- [SeaThru-NeRF: Neural Radiance Fields in Scattering Media](https://mlsanity.com/paper/2304.07743/) (2023)
- [TermiNeRF: Ray Termination Prediction for Efficient Neural Rendering](https://mlsanity.com/paper/2111.03643/) (2021)
- [Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data](https://mlsanity.com/paper/2206.06577/) (2022)
- [Global Transport for Fluid Reconstruction with Learned Self-Supervision](https://mlsanity.com/paper/2104.06031/) (2021)
- [Neural Relightable Participating Media Rendering](https://mlsanity.com/paper/2110.12993/) (2021)
- [Transport-Based Neural Style Transfer for Smoke Simulations](https://mlsanity.com/paper/1905.07442/) (2019)
- [Multi-view polarimetric scattering cloud tomography and retrieval of droplet size](https://mlsanity.com/paper/2005.11423/) (2020)
- [ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields](https://mlsanity.com/paper/2310.13670/) (2023)

## Built on by (most cited first, 2 of 2)

- [Smoke and Mirrors in Causal Downstream Tasks](https://mlsanity.com/paper/2405.17151/) (2024)
- [Success Probability in Multi-View Imaging](https://mlsanity.com/paper/2407.21027/) (2024)

## Abstract (arXiv)

Volumetric phenomena, such as clouds and fog, present a significant challenge for 3D reconstruction systems due to their translucent nature and their complex interactions with light. Conventional techniques for reconstructing scattering volumes rely on controlled setups, limiting practical applications. This paper introduces an approach to reconstructing volumes from a few input stereo pairs. We propose a novel deep learning framework that integrates a deep stereo model with a 3D Convolutional Neural Network (3D CNN) and an advection module, capable of capturing the shape and dynamics of volumes. The stereo depths are used to carve empty space around volumes, providing the 3D CNN with a prior for coping with the lack of input views. Refining our output, the advection module leverages the temporal evolution of the medium, providing a mechanism to infer motion and improve temporal consistency. The efficacy of our system is demonstrated through its ability to estimate density and velocity fields of large-scale volumes, in this case, clouds, from a sparse set of stereo image pairs.