# RT-NeRV: Rethinking Hybrid Neural Representations for Video via Residual Tokenization

arXiv: https://arxiv.org/abs/2403.12401 · HTML: https://mlsanity.com/paper/2403.12401/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context](https://mlsanity.com/paper/2207.08132/) (2022)
- [HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation](https://mlsanity.com/paper/2306.09818/) (2023)
- [FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos](https://mlsanity.com/paper/2212.12294/) (2022)
- [DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos](https://mlsanity.com/paper/2304.06544/) (2023)
- [Towards Scalable Neural Representation for Diverse Videos](https://mlsanity.com/paper/2303.14124/) (2023)
- [PS-NeRV: Patch-wise Stylized Neural Representations for Videos](https://mlsanity.com/paper/2208.03742/) (2022)

## Built on by (most cited first, 7 of 7)

- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [How to Design and Train Your Implicit Neural Representation for Video Compression](https://mlsanity.com/paper/2506.24127/) (2025)
- [Tree-NeRV: A Tree-Structured Neural Representation for Efficient Non-Uniform Video Encoding](https://mlsanity.com/paper/2504.12899/) (2025)
- [GSVR: 2D Gaussian-based Video Representation for 800+ FPS with Hybrid Deformation Field](https://mlsanity.com/paper/2507.05594/) (2025)
- [FANeRV: Frequency Separation and Augmentation based Neural Representation for Video](https://mlsanity.com/paper/2504.06755/) (2025)
- [Range Image-Based Implicit Neural Compression for LiDAR Point Clouds](https://mlsanity.com/paper/2504.17229/) (2025)
- [LRConv-NeRV: Low Rank Convolution for Efficient Neural Video Compression](https://mlsanity.com/paper/2603.18261/) (2026)

## Abstract (arXiv)

Neural Representations for Videos(NeRV) have emerged as a promising paradigm for video compression by representing videos as compact neural networks with efficient decoding. Hybrid NeRV methods further improve reconstruction quality through content adaptive embeddings, but still struggle to preserve fine details at low bitrates. A key limitation is that shallow residual support in formation, although highly beneficial for reconstruction, is costly to transmit in its continuous form and is therefore underutilized. In this paper, we rethink hybrid NeRV and present RT-NeRV, a residual tokenization framework for hybrid neural video representations. The core idea is to discretize shallow residual features and inter-frame residual cues into compact residual tokens, allowing informative reconstruction support to be transmitted efficiently and exploited by the decoder. To this end, we design a residual tokenizer together with a residual-aware codebook learning strategy that improves token utilization and stabilizes training. RT-NeRV can be readily integrated into modern hybrid NeRV hosts, consistently enhancing detail preservation, reconstruction quality, and bitrate quality trade-offs. Extensive experiments on video regression and related restoration tasks show that RT-NeRV outperforms strong hybrid NeRV baselines and remains competitive with recent INR based video compression methods. These results demonstrate that residual tokenization is an effective and complementary direction for advancing hybrid neural video representations