# Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time

arXiv: https://arxiv.org/abs/2309.14022 · HTML: https://mlsanity.com/paper/2309.14022/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Stitch it in Time: GAN-Based Facial Editing of Real Videos](https://mlsanity.com/paper/2201.08361/) (2022)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Deformable Sprites for Unsupervised Video Decomposition](https://mlsanity.com/paper/2204.07151/) (2022)
- [Third Time's the Charm? Image and Video Editing with StyleGAN3](https://mlsanity.com/paper/2201.13433/) (2022)
- [Layered Neural Rendering for Retiming People in Video](https://mlsanity.com/paper/2009.07833/) (2020)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Video Propagation Networks](https://mlsanity.com/paper/1612.05478/) (2016)
- [Learning Pixel Trajectories with Multiscale Contrastive Random Walks](https://mlsanity.com/paper/2201.08379/) (2022)
- [Temporally Consistent Semantic Video Editing](https://mlsanity.com/paper/2206.10590/) (2022)
- [FactorMatte: Redefining Video Matting for Re-Composition Tasks](https://mlsanity.com/paper/2211.02145/) (2022)
- [LayerBuilder: Layer Decomposition for Interactive Image and Video Color Editing](https://mlsanity.com/paper/1701.03754/) (2017)
- [Controllable Attention for Structured Layered Video Decomposition](https://mlsanity.com/paper/1910.11306/) (2019)

## Built on by (most cited first, 5 of 5)

- [Splatter a Video: Video Gaussian Representation for Versatile Processing](https://mlsanity.com/paper/2406.13870/) (2024)
- [VeGaS: Video Gaussian Splatting](https://mlsanity.com/paper/2411.11024/) (2024)
- [NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing](https://mlsanity.com/paper/2406.06523/) (2024)
- [RNA: Video Editing with ROI-based Neural Atlas](https://mlsanity.com/paper/2410.07600/) (2024)
- [HyperNVD: Accelerating Neural Video Decomposition via Hypernetworks](https://mlsanity.com/paper/2503.17276/) (2025)

## Abstract (arXiv)

We present a video decomposition method that facilitates layer-based editing of videos with spatiotemporally varying lighting and motion effects. Our neural model decomposes an input video into multiple layered representations, each comprising a 2D texture map, a mask for the original video, and a multiplicative residual characterizing the spatiotemporal variations in lighting conditions. A single edit on the texture maps can be propagated to the corresponding locations in the entire video frames while preserving other contents' consistencies. Our method efficiently learns the layer-based neural representations of a 1080p video in 25s per frame via coordinate hashing and allows real-time rendering of the edited result at 71 fps on a single GPU. Qualitatively, we run our method on various videos to show its effectiveness in generating high-quality editing effects. Quantitatively, we propose to adopt feature-tracking evaluation metrics for objectively assessing the consistency of video editing. Project page: https://lightbulb12294.github.io/hashing-nvd/