# Deformable Neural Radiance Fields using RGB and Event Cameras

arXiv: https://arxiv.org/abs/2309.08416 · HTML: https://mlsanity.com/paper/2309.08416/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [ICON: Implicit Clothed humans Obtained from Normals](https://mlsanity.com/paper/2112.09127/) (2021)
- [Monocular Dynamic View Synthesis: A Reality Check](https://mlsanity.com/paper/2210.13445/) (2022)
- [NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields](https://mlsanity.com/paper/2210.15947/) (2022)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Event-based Stereo Visual Odometry](https://mlsanity.com/paper/2007.15548/) (2020)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [EventNeRF: Neural Radiance Fields from a Single Colour Event Camera](https://mlsanity.com/paper/2206.11896/) (2022)
- [Deep Implicit Templates for 3D Shape Representation](https://mlsanity.com/paper/2011.14565/) (2020)
- [E-NeRF: Neural Radiance Fields from a Moving Event Camera](https://mlsanity.com/paper/2208.11300/) (2022)
- [Semi-Dense 3D Reconstruction with a Stereo Event Camera](https://mlsanity.com/paper/1807.07429/) (2018)
- [Ev-NeRF: Event Based Neural Radiance Field](https://mlsanity.com/paper/2206.12455/) (2022)
- [EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream](https://mlsanity.com/paper/2012.06475/) (2020)
- [CED: Color Event Camera Dataset](https://mlsanity.com/paper/1904.10772/) (2019)
- [EVIMO2: An Event Camera Dataset for Motion Segmentation, Optical Flow, Structure from Motion, and Visual Inertial Odometry in Indoor Scenes with Monocular or Stereo Algorithms](https://mlsanity.com/paper/2205.03467/) (2022)
- [Event-based Non-Rigid Reconstruction from Contours](https://mlsanity.com/paper/2210.06270/) (2022)
- [E3D: Event-Based 3D Shape Reconstruction](https://mlsanity.com/paper/2012.05214/) (2020)
- [EvAC3D: From Event-based Apparent Contours to 3D Models via Continuous Visual Hulls](https://mlsanity.com/paper/2304.05296/) (2023)

## Built on by (most cited first, 30 of 31)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion](https://mlsanity.com/paper/2406.02972/) (2024)
- [Implicit Event-RGBD Neural SLAM](https://mlsanity.com/paper/2311.11013/) (2023)
- [Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment](https://mlsanity.com/paper/2406.14360/) (2024)
- [EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting](https://mlsanity.com/paper/2410.15392/) (2024)
- [Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions](https://mlsanity.com/paper/2409.17988/) (2024)
- [Time-Efficient Light-Field Acquisition Using Coded Aperture and Events](https://mlsanity.com/paper/2403.07244/) (2024)
- [Entity-NeRF: Detecting and Removing Moving Entities in Urban Scenes](https://mlsanity.com/paper/2403.16141/) (2024)
- [E3NeRF: Efficient Event-Enhanced Neural Radiance Fields from Blurry Images](https://mlsanity.com/paper/2408.01840/) (2024)
- [Spike-NeRF: Neural Radiance Field Based On Spike Camera](https://mlsanity.com/paper/2403.16410/) (2024)
- [SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion](https://mlsanity.com/paper/2407.10062/) (2024)
- [GS2E: Gaussian Splatting is an Effective Data Generator for Event Stream Generation](https://mlsanity.com/paper/2505.15287/) (2025)
- [SpikeGS: Reconstruct 3D scene via fast-moving bio-inspired sensors](https://mlsanity.com/paper/2407.03771/) (2024)
- [A Survey of 3D Reconstruction with Event Cameras](https://mlsanity.com/paper/2505.08438/) (2025)
- [Continuous Pose for Monocular Cameras in Neural Implicit Representation](https://mlsanity.com/paper/2311.17119/) (2023)
- [LSE-NeRF: Learning Sensor Modeling Errors for Deblured Neural Radiance Fields with RGB-Event Stereo](https://mlsanity.com/paper/2409.06104/) (2024)
- [Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams](https://mlsanity.com/paper/2412.06770/) (2024)
- [USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting](https://mlsanity.com/paper/2411.10504/) (2024)
- [Event-boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2411.16180/) (2024)
- [Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Survey](https://mlsanity.com/paper/2509.09971/) (2025)
- [DEGS: Deformable Event-based 3D Gaussian Splatting from RGB and Event Stream](https://mlsanity.com/paper/2510.07752/) (2025)
- [Ev4DGS: Novel-view Rendering of Non-Rigid Objects from Monocular Event Streams](https://mlsanity.com/paper/2510.11717/) (2025)
- [Few-shot Implicit Function Generation via Equivariance](https://mlsanity.com/paper/2501.01601/) (2025)
- [ExFMan: Rendering 3D Dynamic Humans with Hybrid Monocular Blurry Frames and Events](https://mlsanity.com/paper/2409.14103/) (2024)
- [SaENeRF: Suppressing Artifacts in Event-based Neural Radiance Fields](https://mlsanity.com/paper/2504.16389/) (2025)
- [STD-GS: Exploring Frame-Event Interaction for SpatioTemporal-Disentangled Gaussian Splatting to Reconstruct High-Dynamic Scene](https://mlsanity.com/paper/2506.23157/) (2025)
- [AsyncEvGS: Asynchronous Event-Assisted Gaussian Splatting for Handheld Motion-Blurred Scenes](https://mlsanity.com/paper/2605.07192/) (2026)
- [PS4PRO: Pixel-to-pixel Supervision for Photorealistic Rendering and Optimization](https://mlsanity.com/paper/2505.22616/) (2025)
- [ERF-GS: Reconstructing Fast Motion from Disjoint Event-RGB Viewpoints](https://mlsanity.com/paper/2608.08531/) (2026)

## Abstract (arXiv)

Modeling Neural Radiance Fields for fast-moving deformable objects from visual data alone is a challenging problem. A major issue arises due to the high deformation and low acquisition rates. To address this problem, we propose to use event cameras that offer very fast acquisition of visual change in an asynchronous manner. In this work, we develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and calibrated sparse RGB frames. In our setup, the camera pose at the individual events required to integrate them into the radiance fields remains unknown. Our method jointly optimizes these poses and the radiance field. This happens efficiently by leveraging the collection of events at once and actively sampling the events during learning. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method over the state-of-the-art and the compared baseline. This shows a promising direction for modeling deformable neural radiance fields in real-world dynamic scenes.