# Real-Time Neural Materials using Block-Compressed Features

arXiv: https://arxiv.org/abs/2311.16121 · HTML: https://mlsanity.com/paper/2311.16121/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation](https://mlsanity.com/paper/2204.05735/) (2022)
- [NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions](https://mlsanity.com/paper/2309.15426/) (2023)
- [Binary Radiance Fields](https://mlsanity.com/paper/2306.07581/) (2023)
- [Neural BRDF Representation and Importance Sampling](https://mlsanity.com/paper/2102.05963/) (2021)
- [Random-Access Neural Compression of Material Textures](https://mlsanity.com/paper/2305.17105/) (2023)
- [Real-Time Neural Appearance Models](https://mlsanity.com/paper/2305.02678/) (2023)
- [Neural Shadow Mapping](https://mlsanity.com/paper/2301.05262/) (2023)

## Built on by (most cited first, 4 of 4)

- [Neural Dynamic GI: Random-Access Neural Compression for Temporal Lightmaps in Dynamic Lighting Environments](https://mlsanity.com/paper/2604.12625/) (2026)
- [PEPS: Positional Encoding Projected Sampling -- Extended](https://mlsanity.com/paper/2604.24167/) (2026)
- [A Hybrid Neural-Microfacet BRDF Model for Real-Time Rendering](https://mlsanity.com/paper/2608.09604/) (2026)
- [Thread-Efficient Decoding for Neural Texture Compression](https://mlsanity.com/paper/2608.27888/) (2026)

## Abstract (arXiv)

Neural materials typically consist of a collection of neural features along with a decoder network. The main challenge in integrating such models in real-time rendering pipelines lies in the large size required to store their features in GPU memory and the complexity of evaluating the network efficiently. We present a neural material model whose features and decoder are specifically designed to be used in real-time rendering pipelines. Our framework leverages hardware-based block compression (BC) texture formats to store the learned features and trains the model to output the material information continuously in space and scale. To achieve this, we organize the features in a block-based manner and emulate BC6 decompression during training, making it possible to export them as regular BC6 textures. This structure allows us to use high resolution features while maintaining a low memory footprint. Consequently, this enhances our model's overall capability, enabling the use of a lightweight and simple decoder architecture that can be evaluated directly in a shader. Furthermore, since the learned features can be decoded continuously, it allows for random uv sampling and smooth transition between scales without needing any subsequent filtering. As a result, our neural material has a small memory footprint, can be decoded extremely fast adding a minimal computational overhead to the rendering pipeline.