# HyperNVD: Accelerating Neural Video Decomposition via Hypernetworks

arXiv: https://arxiv.org/abs/2503.17276 · HTML: https://mlsanity.com/paper/2503.17276/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 29 of 29)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing](https://mlsanity.com/paper/2310.05922/) (2023)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [Adversarial Generation of Continuous Images](https://mlsanity.com/paper/2011.12026/) (2020)
- [CoDeF: Content Deformation Fields for Temporally Consistent Video Processing](https://mlsanity.com/paper/2308.07926/) (2023)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Stylizing 3D Scene via Implicit Representation and HyperNetwork](https://mlsanity.com/paper/2105.13016/) (2021)
- [Deformable Sprites for Unsupervised Video Decomposition](https://mlsanity.com/paper/2204.07151/) (2022)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Generalizable Implicit Neural Representations via Instance Pattern Composers](https://mlsanity.com/paper/2211.13223/) (2022)
- [MarioNette: Self-Supervised Sprite Learning](https://mlsanity.com/paper/2104.14553/) (2021)
- [TarViS: A Unified Approach for Target-based Video Segmentation](https://mlsanity.com/paper/2301.02657/) (2023)
- [Neural Image Representations for Multi-Image Fusion and Layer Separation](https://mlsanity.com/paper/2108.01199/) (2021)
- [HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork](https://mlsanity.com/paper/2306.06093/) (2023)
- [Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics](https://mlsanity.com/paper/2408.02672/) (2024)
- [FactorMatte: Redefining Video Matting for Re-Composition Tasks](https://mlsanity.com/paper/2211.02145/) (2022)
- [Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time](https://mlsanity.com/paper/2309.14022/) (2023)
- [RNA: Video Editing with ROI-based Neural Atlas](https://mlsanity.com/paper/2410.07600/) (2024)

## Built on by (most cited first, 1 of 1)

- [HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks](https://mlsanity.com/paper/2512.12544/) (2025)

## Abstract (arXiv)

Decomposing a video into a layer-based representation is crucial for easy video editing for the creative industries, as it enables independent editing of specific layers. Existing video-layer decomposition models rely on implicit neural representations (INRs) trained independently for each video, making the process time-consuming when applied to new videos. Noticing this limitation, we propose a meta-learning strategy to learn a generic video decomposition model to speed up the training on new videos. Our model is based on a hypernetwork architecture which, given a video-encoder embedding, generates the parameters for a compact INR-based neural video decomposition model. Our strategy mitigates the problem of single-video overfitting and, importantly, shortens the convergence of video decomposition on new, unseen videos. Our code is available at: https://hypernvd.github.io/