# Scalable Neural Video Representations with Learnable Positional Features

arXiv: https://arxiv.org/abs/2210.06823 · HTML: https://mlsanity.com/paper/2210.06823/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 63 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 40)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://mlsanity.com/paper/2202.10571/) (2022)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)

## Built on by (most cited first, 29 of 29)

- [Video Probabilistic Diffusion Models in Projected Latent Space](https://mlsanity.com/paper/2302.07685/) (2023)
- [Spatial Functa: Scaling Functa to ImageNet Classification and Generation](https://mlsanity.com/paper/2302.03130/) (2023)
- [NIRVANA: Neural Implicit Representations of Videos with Adaptive Networks and Autoregressive Patch-wise Modeling](https://mlsanity.com/paper/2212.14593/) (2022)
- [FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos](https://mlsanity.com/paper/2212.12294/) (2022)
- [Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition](https://mlsanity.com/paper/2403.14148/) (2024)
- [Binary Radiance Fields](https://mlsanity.com/paper/2306.07581/) (2023)
- [DINER: Disorder-Invariant Implicit Neural Representation](https://mlsanity.com/paper/2211.07871/) (2022)
- [Disorder-invariant Implicit Neural Representation](https://mlsanity.com/paper/2304.00837/) (2023)
- [DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes](https://mlsanity.com/paper/2403.15679/) (2024)
- [Modality-Agnostic Variational Compression of Implicit Neural Representations](https://mlsanity.com/paper/2301.09479/) (2023)
- [MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation](https://mlsanity.com/paper/2502.12632/) (2025)
- [Learning Large-scale Neural Fields via Context Pruned Meta-Learning](https://mlsanity.com/paper/2302.00617/) (2023)
- [Fast Encoding and Decoding for Implicit Video Representation](https://mlsanity.com/paper/2409.19429/) (2024)
- [RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations](https://mlsanity.com/paper/2309.17182/) (2023)
- [Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics](https://mlsanity.com/paper/2408.02672/) (2024)
- [BF-STVSR: B-Splines and Fourier-Best Friends for High Fidelity Spatial-Temporal Video Super-Resolution](https://mlsanity.com/paper/2501.11043/) (2025)
- [Implicit-explicit Integrated Representations for Multi-view Video Compression](https://mlsanity.com/paper/2311.17350/) (2023)
- [In Search of a Data Transformation That Accelerates Neural Field Training](https://mlsanity.com/paper/2311.17094/) (2023)
- [EVOS: Efficient Implicit Neural Training via EVOlutionary Selector](https://mlsanity.com/paper/2412.10153/) (2024)
- [Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction](https://mlsanity.com/paper/2411.14762/) (2024)
- [Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions](https://mlsanity.com/paper/2401.10217/) (2024)
- [How to Design and Train Your Implicit Neural Representation for Video Compression](https://mlsanity.com/paper/2506.24127/) (2025)
- [Predicting the Encoding Error of SIRENs](https://mlsanity.com/paper/2410.21645/) (2024)
- [Efficient Neural Video Representation with Temporally Coherent Modulation](https://mlsanity.com/paper/2505.00335/) (2025)
- [NTK-Guided Implicit Neural Teaching](https://mlsanity.com/paper/2511.15487/) (2025)
- [FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fields](https://mlsanity.com/paper/2508.06301/) (2025)
- [Implicit Neural Representation Facilitates Unified Universal Vision Encoding](https://mlsanity.com/paper/2601.14256/) (2026)
- [MNeRV: A Multilayer Neural Representation for Videos](https://mlsanity.com/paper/2407.07347/) (2024)
- [CoANeRV: Coordinate-Aware Token-Space Neural Video Representation](https://mlsanity.com/paper/2608.13938/) (2026)

## Abstract (arXiv)

Succinct representation of complex signals using coordinate-based neural representations (CNRs) has seen great progress, and several recent efforts focus on extending them for handling videos. Here, the main challenge is how to (a) alleviate a compute-inefficiency in training CNRs to (b) achieve high-quality video encoding while (c) maintaining the parameter-efficiency. To meet all requirements (a), (b), and (c) simultaneously, we propose neural video representations with learnable positional features (NVP), a novel CNR by introducing "learnable positional features" that effectively amortize a video as latent codes. Specifically, we first present a CNR architecture based on designing 2D latent keyframes to learn the common video contents across each spatio-temporal axis, which dramatically improves all of those three requirements. Then, we propose to utilize existing powerful image and video codecs as a compute-/memory-efficient compression procedure of latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow34.57 (measured with the PSNR metric), even using >8 times fewer parameters. We also show intriguing properties of NVP, e.g., video inpainting, video frame interpolation, etc.