# DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos

arXiv: https://arxiv.org/abs/2304.06544 · HTML: https://mlsanity.com/paper/2304.06544/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 64 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://mlsanity.com/paper/2202.10571/) (2022)
- [Deblur-NeRF: Neural Radiance Fields from Blurry Images](https://mlsanity.com/paper/2111.14292/) (2021)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Implicit Neural Representations for Image Compression](https://mlsanity.com/paper/2112.04267/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)

## Built on by (most cited first, 30 of 42)

- [Implicit Neural Representation in Medical Imaging: A Comparative Survey](https://mlsanity.com/paper/2307.16142/) (2023)
- [Boosting Neural Representations for Videos with a Conditional Decoder](https://mlsanity.com/paper/2402.18152/) (2024)
- [NVRC: Neural Video Representation Compression](https://mlsanity.com/paper/2409.07414/) (2024)
- [DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes](https://mlsanity.com/paper/2403.15679/) (2024)
- [Neural Video Compression with Context Modulation](https://mlsanity.com/paper/2505.14541/) (2025)
- [PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos](https://mlsanity.com/paper/2404.08921/) (2024)
- [GIViC: Generative Implicit Video Compression](https://mlsanity.com/paper/2503.19604/) (2025)
- [VeGaS: Video Gaussian Splatting](https://mlsanity.com/paper/2411.11024/) (2024)
- [SNeRV: Spectra-preserving Neural Representation for Video](https://mlsanity.com/paper/2501.01681/) (2025)
- [Implicit-explicit Integrated Representations for Multi-view Video Compression](https://mlsanity.com/paper/2311.17350/) (2023)
- [GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting](https://mlsanity.com/paper/2503.04333/) (2025)
- [Augmented Deep Contexts for Spatially Embedded Video Coding](https://mlsanity.com/paper/2505.05309/) (2025)
- [RT-NeRV: Rethinking Hybrid Neural Representations for Video via Residual Tokenization](https://mlsanity.com/paper/2403.12401/) (2024)
- [Superpixel-informed Implicit Neural Representation for Multi-Dimensional Data](https://mlsanity.com/paper/2411.11356/) (2024)
- [FCNR: Fast Compressive Neural Representation of Visualization Images](https://mlsanity.com/paper/2407.16369/) (2024)
- [Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions](https://mlsanity.com/paper/2401.10217/) (2024)
- [Prediction and Reference Quality Adaptation for Learned Video Compression](https://mlsanity.com/paper/2406.14118/) (2024)
- [Real-Time Neural Video Compression with Unified Intra and Inter Coding](https://mlsanity.com/paper/2510.14431/) (2025)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)
- [Neural Video Representation for Redundancy Reduction and Consistency Preservation](https://mlsanity.com/paper/2409.18497/) (2024)
- [How to Design and Train Your Implicit Neural Representation for Video Compression](https://mlsanity.com/paper/2506.24127/) (2025)
- [SGI: Structured 2D Gaussians for Efficient and Compact Large Image Representation](https://mlsanity.com/paper/2603.07789/) (2026)
- [High-Frequency Enhanced Hybrid Neural Representation for Video Compression](https://mlsanity.com/paper/2411.06685/) (2024)
- [Bias for Action: Video Implicit Neural Representations with Bias Modulation](https://mlsanity.com/paper/2501.09277/) (2025)
- [CANeRV: Content Adaptive Neural Representation for Video Compression](https://mlsanity.com/paper/2502.06181/) (2025)
- [Tree-NeRV: A Tree-Structured Neural Representation for Efficient Non-Uniform Video Encoding](https://mlsanity.com/paper/2504.12899/) (2025)
- [Efficient Neural Video Representation with Temporally Coherent Modulation](https://mlsanity.com/paper/2505.00335/) (2025)
- [STGV: Spatio-Temporal Hash Encoding for Gaussian-based Video Representation](https://mlsanity.com/paper/2604.10910/) (2026)
- [Implicit Neural Representation Facilitates Unified Universal Vision Encoding](https://mlsanity.com/paper/2601.14256/) (2026)
- [TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos](https://mlsanity.com/paper/2602.16711/) (2026)

## Abstract (arXiv)

Existing implicit neural representation (INR) methods do not fully exploit spatiotemporal redundancies in videos. Index-based INRs ignore the content-specific spatial features and hybrid INRs ignore the contextual dependency on adjacent frames, leading to poor modeling capability for scenes with large motion or dynamics. We analyze this limitation from the perspective of function fitting and reveal the importance of frame difference. To use explicit motion information, we propose Difference Neural Representation for Videos (DNeRV), which consists of two streams for content and frame difference. We also introduce a collaborative content unit for effective feature fusion. We test DNeRV for video compression, inpainting, and interpolation. DNeRV achieves competitive results against the state-of-the-art neural compression approaches and outperforms existing implicit methods on downstream inpainting and interpolation for 960 × 1920 videos.