# Dynamic Mesh-Aware Radiance Fields

arXiv: https://arxiv.org/abs/2309.04581 · HTML: https://mlsanity.com/paper/2309.04581/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 93 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 30 of 56)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)

## Built on by (most cited first, 23 of 23)

- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2405.17429/) (2024)
- [Gaussian Splashing: Unified Particles for Versatile Motion Synthesis and Rendering](https://mlsanity.com/paper/2401.15318/) (2024)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Models](https://mlsanity.com/paper/2312.08563/) (2023)
- [Articulated Kinematics Distillation from Video Diffusion Models](https://mlsanity.com/paper/2504.01204/) (2025)
- [Neural Radiance Field-based Visual Rendering: A Comprehensive Review](https://mlsanity.com/paper/2404.00714/) (2024)
- [Learning rigid-body simulators over implicit shapes for large-scale scenes and vision](https://mlsanity.com/paper/2405.14045/) (2024)
- [3D Points Splatting for Real-Time Dynamic Hand Reconstruction](https://mlsanity.com/paper/2312.13770/) (2023)
- [MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting](https://mlsanity.com/paper/2406.01593/) (2024)
- [Dynamic Scene Reconstruction: Recent Advance in Real-time Rendering and Streaming](https://mlsanity.com/paper/2503.08166/) (2025)
- [Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling](https://mlsanity.com/paper/2406.03723/) (2024)
- [Radiance Fields in XR: A Survey on How Radiance Fields are Envisioned and Addressed for XR Research](https://mlsanity.com/paper/2508.04326/) (2025)
- [A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields](https://mlsanity.com/paper/2310.05837/) (2023)
- [Scaling Face Interaction Graph Networks to Real World Scenes](https://mlsanity.com/paper/2401.11985/) (2024)
- [PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animation](https://mlsanity.com/paper/2506.01091/) (2025)
- [Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2404.09715/) (2024)
- [Towards Realistic Example-based Modeling via 3D Gaussian Stitching](https://mlsanity.com/paper/2408.15708/) (2024)
- [PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenes](https://mlsanity.com/paper/2512.24986/) (2025)
- [Voxel-Mesh Hybrid Representation for Real-Time View Synthesis](https://mlsanity.com/paper/2403.06505/) (2024)
- [InfoGaussian: Structure-Aware Dynamic Gaussians through Lightweight Information Shaping](https://mlsanity.com/paper/2406.05897/) (2024)
- [3D Dynamic Fluid Assets from Single-View Videos with Generative Gaussian Splatting](https://mlsanity.com/paper/2503.00868/) (2025)
- [PaMoSplat: Part-Aware Motion-Guided Gaussian Splatting for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2605.10307/) (2026)

## Abstract (arXiv)

Embedding polygonal mesh assets within photorealistic Neural Radience Fields (NeRF) volumes, such that they can be rendered and their dynamics simulated in a physically consistent manner with the NeRF, is under-explored from the system perspective of integrating NeRF into the traditional graphics pipeline. This paper designs a two-way coupling between mesh and NeRF during rendering and simulation. We first review the light transport equations for both mesh and NeRF, then distill them into an efficient algorithm for updating radiance and throughput along a cast ray with an arbitrary number of bounces. To resolve the discrepancy between the linear color space that the path tracer assumes and the sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range (HDR) images. We also present a strategy to estimate light sources and cast shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric formulation can be efficiently integrated with a high-performance physics simulator that supports cloth, rigid and soft bodies. The full rendering and simulation system can be run on a GPU at interactive rates. We show that a hybrid system approach outperforms alternatives in visual realism for mesh insertion, because it allows realistic light transport from volumetric NeRF media onto surfaces, which affects the appearance of reflective/refractive surfaces and illumination of diffuse surfaces informed by the dynamic scene.