# Inferring Hybrid Neural Fluid Fields from Videos

arXiv: https://arxiv.org/abs/2312.06561 · HTML: https://mlsanity.com/paper/2312.06561/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural Radiance Flow for 4D View Synthesis and Video Processing](https://mlsanity.com/paper/2012.09790/) (2020)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [Accelerating Eulerian Fluid Simulation With Convolutional Networks](https://mlsanity.com/paper/1607.03597/) (2016)
- [Deep Fluids: A Generative Network for Parameterized Fluid Simulations](https://mlsanity.com/paper/1806.02071/) (2018)
- [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021)
- [Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow](https://mlsanity.com/paper/1802.10123/) (2018)
- [Object Scene Representation Transformer](https://mlsanity.com/paper/2206.06922/) (2022)
- [Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data](https://mlsanity.com/paper/2206.06577/) (2022)
- [Differentiable Physics Simulation of Dynamics-Augmented Neural Objects](https://mlsanity.com/paper/2210.09420/) (2022)
- [NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields](https://mlsanity.com/paper/2203.01762/) (2022)
- [Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics](https://mlsanity.com/paper/2210.06036/) (2022)
- [Global Transport for Fluid Reconstruction with Learned Self-Supervision](https://mlsanity.com/paper/2104.06031/) (2021)
- [Learning Vortex Dynamics for Fluid Inference and Prediction](https://mlsanity.com/paper/2301.11494/) (2023)
- [Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition](https://mlsanity.com/paper/2303.06138/) (2023)
- [Multi-Object Manipulation via Object-Centric Neural Scattering Functions](https://mlsanity.com/paper/2306.08748/) (2023)
- [Coupled Fluid Density and Motion from Single Views](https://mlsanity.com/paper/1806.06613/) (2018)

## Built on by (most cited first, 18 of 18)

- [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Video](https://mlsanity.com/paper/2503.04720/) (2025)
- [Physics-Informed Learning of Characteristic Trajectories for Smoke Reconstruction](https://mlsanity.com/paper/2407.09679/) (2024)
- [Physics-Informed Deformable Gaussian Splatting: Towards Unified Constitutive Laws for Time-Evolving Material Field](https://mlsanity.com/paper/2511.06299/) (2025)
- [Representing Flow Fields with Divergence-Free Kernels for Reconstruction](https://mlsanity.com/paper/2504.01913/) (2025)
- [Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Model](https://mlsanity.com/paper/2412.13897/) (2024)
- [ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimation](https://mlsanity.com/paper/2506.05317/) (2025)
- [Learning an Implicit Physics Model for Image-based Fluid Simulation](https://mlsanity.com/paper/2508.08254/) (2025)
- [LagrangianSplats: Divergence-Free Transport of Gaussian Primitives for Fluid Reconstruction](https://mlsanity.com/paper/2605.09299/) (2026)
- [WildSmoke: Ready-to-Use Dynamic 3D Smoke Assets from a Single Video in the Wild](https://mlsanity.com/paper/2509.11114/) (2025)
- [SurfPhase: 3D Interfacial Dynamics in Two-Phase Flows from Sparse Videos](https://mlsanity.com/paper/2602.11154/) (2026)
- [ReMatching Dynamic Reconstruction Flow](https://mlsanity.com/paper/2411.00705/) (2024)
- [SmokeSVD: Smoke Reconstruction from A Single View via Progressive Novel View Synthesis and Refinement with Diffusion Models](https://mlsanity.com/paper/2507.12156/) (2025)
- [From Video-to-PDE: Data-Driven Discovery of Nonlinear Dye Plume Dynamics](https://mlsanity.com/paper/2605.04535/) (2026)
- [CausalGS: Learning Physical Causality of 3D Dynamic Scenes with Gaussian Representations](https://mlsanity.com/paper/2605.10586/) (2026)
- [SplashSplat: Reconstructing Splashing Liquids from Real-World Multi-View Videos](https://mlsanity.com/paper/2609.20818/) (2026)

## Abstract (arXiv)

We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https://kovenyu.com/HyFluid/