# Learning Kernel-Modulated Neural Representation for Efficient Light Field Compression

arXiv: https://arxiv.org/abs/2307.06143 · HTML: https://mlsanity.com/paper/2307.06143/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 81 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [Implicit Neural Representations for Image Compression](https://mlsanity.com/paper/2112.04267/) (2021)
- [Training with Quantization Noise for Extreme Model Compression](https://mlsanity.com/paper/2004.07320/) (2020)
- [A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1804.03294/) (2018)
- [Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks](https://mlsanity.com/paper/1810.03982/) (2018)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Disentangling Light Fields for Super-Resolution and Disparity Estimation](https://mlsanity.com/paper/2202.10603/) (2022)
- [Perceptual Learned Video Compression with Recurrent Conditional GAN](https://mlsanity.com/paper/2109.03082/) (2021)
- [TransCut: Transparent Object Segmentation from a Light-Field Image](https://mlsanity.com/paper/1511.06853/) (2015)
- [Detail-Preserving Transformer for Light Field Image Super-Resolution](https://mlsanity.com/paper/2201.00346/) (2022)
- [DCFNet: Deep Neural Network with Decomposed Convolutional Filters](https://mlsanity.com/paper/1802.04145/) (2018)
- [Occlusion-Aware Cost Constructor for Light Field Depth Estimation](https://mlsanity.com/paper/2203.01576/) (2022)
- [Looking Through the Glass: Neural Surface Reconstruction Against High Specular Reflections](https://mlsanity.com/paper/2304.08706/) (2023)
- [JAWS: Just A Wild Shot for Cinematic Transfer in Neural Radiance Fields](https://mlsanity.com/paper/2303.15427/) (2023)
- [Pseudo Sequence based 2-D hierarchical reference structure for Light-Field Image Compression](https://mlsanity.com/paper/1612.07309/) (2016)
- [Light Field Compression with Disparity Guided Sparse Coding based on Structural Key Views](https://mlsanity.com/paper/1610.03684/) (2016)
- [Geometry-Aware Graph Transforms for Light Field Compact Representation](https://mlsanity.com/paper/1903.03556/) (2019)
- [Content-aware Warping for View Synthesis](https://mlsanity.com/paper/2201.09023/) (2022)

## Built on by (most cited first, 2 of 2)

- [LFIC-DRASC: Deep Light Field Image Compression Using Disentangled Representation and Asymmetrical Strip Convolution](https://mlsanity.com/paper/2409.11711/) (2024)
- [SANR: Scene-Aware Neural Representation for Light Field Image Compression with Rate-Distortion Optimization](https://mlsanity.com/paper/2510.15775/) (2025)

## Abstract (arXiv)

Light field is a type of image data that captures the 3D scene information by recording light rays emitted from a scene at various orientations. It offers a more immersive perception than classic 2D images but at the cost of huge data volume. In this paper, we draw inspiration from the visual characteristics of Sub-Aperture Images (SAIs) of light field and design a compact neural network representation for the light field compression task. The network backbone takes randomly initialized noise as input and is supervised on the SAIs of the target light field. It is composed of two types of complementary kernels: descriptive kernels (descriptors) that store scene description information learned during training, and modulatory kernels (modulators) that control the rendering of different SAIs from the queried perspectives. To further enhance compactness of the network meanwhile retain high quality of the decoded light field, we accordingly introduce modulator allocation and kernel tensor decomposition mechanisms, followed by non-uniform quantization and lossless entropy coding techniques, to finally form an efficient compression pipeline. Extensive experiments demonstrate that our method outperforms other state-of-the-art (SOTA) methods by a significant margin in the light field compression task. Moreover, after aligning descriptors, the modulators learned from one light field can be transferred to new light fields for rendering dense views, indicating a potential solution for view synthesis task.