# Flow supervision for Deformable NeRF

arXiv: https://arxiv.org/abs/2303.16333 · HTML: https://mlsanity.com/paper/2303.16333/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [Neural Radiance Flow for 4D View Synthesis and Video Processing](https://mlsanity.com/paper/2012.09790/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Monocular Dynamic View Synthesis: A Reality Check](https://mlsanity.com/paper/2210.13445/) (2022)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [Neural Volumes: Learning Dynamic Renderable Volumes from Images](https://mlsanity.com/paper/1906.07751/) (2019)
- [Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera](https://mlsanity.com/paper/2004.01294/) (2020)
- [DeRF: Decomposed Radiance Fields](https://mlsanity.com/paper/2011.12490/) (2020)
- [D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video](https://mlsanity.com/paper/2205.15838/) (2022)
- [Neural Scene Flow Prior](https://mlsanity.com/paper/2111.01253/) (2021)
- [Neural Trajectory Fields for Dynamic Novel View Synthesis](https://mlsanity.com/paper/2105.05994/) (2021)
- [Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes](https://mlsanity.com/paper/1904.11112/) (2019)
- [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022)

## Built on by (most cited first, 30 of 42)

- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation](https://mlsanity.com/paper/2403.12365/) (2024)
- [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024)
- [GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis](https://mlsanity.com/paper/2312.11458/) (2023)
- [PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos](https://mlsanity.com/paper/2503.17973/) (2025)
- [4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models](https://mlsanity.com/paper/2406.07472/) (2024)
- [Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction](https://mlsanity.com/paper/2312.01196/) (2023)
- [CoGS: Controllable Gaussian Splatting](https://mlsanity.com/paper/2312.05664/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [ResFields: Residual Neural Fields for Spatiotemporal Signals](https://mlsanity.com/paper/2309.03160/) (2023)
- [Diffusion Priors for Dynamic View Synthesis from Monocular Videos](https://mlsanity.com/paper/2401.05583/) (2024)
- [7DGS: Unified Spatial-Temporal-Angular Gaussian Splatting](https://mlsanity.com/paper/2503.07946/) (2025)
- [Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras](https://mlsanity.com/paper/2312.07423/) (2023)
- [DyBluRF: Dynamic Neural Radiance Fields from Blurry Monocular Video](https://mlsanity.com/paper/2403.10103/) (2024)
- [DaReNeRF: Direction-aware Representation for Dynamic Scenes](https://mlsanity.com/paper/2403.02265/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [Sync4D: Video Guided Controllable Dynamics for Physics-Based 4D Generation](https://mlsanity.com/paper/2405.16849/) (2024)
- [HiMo: High-Speed Objects Motion Compensation in Point Clouds](https://mlsanity.com/paper/2503.00803/) (2025)
- [HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scene](https://mlsanity.com/paper/2506.09518/) (2025)
- [TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction](https://mlsanity.com/paper/2411.11941/) (2024)
- [MASIV: Toward Material-Agnostic System Identification from Videos](https://mlsanity.com/paper/2508.01112/) (2025)
- [Entity-NeRF: Detecting and Removing Moving Entities in Urban Scenes](https://mlsanity.com/paper/2403.16141/) (2024)
- [DASH: 4D Hash Encoding with Self-Supervised Decomposition for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2507.19141/) (2025)
- [Dynamic Neural Radiance Field From Defocused Monocular Video](https://mlsanity.com/paper/2407.05586/) (2024)
- [Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules](https://mlsanity.com/paper/2412.04457/) (2024)
- [Invertible Neural Warp for NeRF](https://mlsanity.com/paper/2407.12354/) (2024)
- [Advances in Radiance Field for Dynamic Scene: From Neural Field to Gaussian Field](https://mlsanity.com/paper/2505.10049/) (2025)
- [ReconPhys: Reconstruct Appearance and Physical Attributes from Single Video](https://mlsanity.com/paper/2604.07882/) (2026)
- [Regularizing Dynamic Radiance Fields with Kinematic Fields](https://mlsanity.com/paper/2407.14059/) (2024)
- [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025)

## Abstract (arXiv)

In this paper we present a new method for deformable NeRF that can directly use optical flow as supervision. We overcome the major challenge with respect to the computationally inefficiency of enforcing the flow constraints to the backward deformation field, used by deformable NeRFs. Specifically, we show that inverting the backward deformation function is actually not needed for computing scene flows between frames. This insight dramatically simplifies the problem, as one is no longer constrained to deformation functions that can be analytically inverted. Instead, thanks to the weak assumptions required by our derivation based on the inverse function theorem, our approach can be extended to a broad class of commonly used backward deformation field. We present results on monocular novel view synthesis with rapid object motion, and demonstrate significant improvements over baselines without flow supervision.