# Multi-Head ReLU Implicit Neural Representation Networks

arXiv: https://arxiv.org/abs/2110.03448 · HTML: https://mlsanity.com/paper/2110.03448/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Modulated Periodic Activations for Generalizable Local Functional Representations](https://mlsanity.com/paper/2104.03960/) (2021)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)

## Built on by (most cited first, 10 of 10)

- [Implicit Neural Representation in Medical Imaging: A Comparative Survey](https://mlsanity.com/paper/2307.16142/) (2023)
- [TINC: Tree-structured Implicit Neural Compression](https://mlsanity.com/paper/2211.06689/) (2022)
- [SCI: A Spectrum Concentrated Implicit Neural Compression for Biomedical Data](https://mlsanity.com/paper/2209.15180/) (2022)
- [Towards Croppable Implicit Neural Representations](https://mlsanity.com/paper/2409.19472/) (2024)
- [A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representations](https://mlsanity.com/paper/2502.00869/) (2025)
- [Encoding Semantic Priors into the Weights of Implicit Neural Representation](https://mlsanity.com/paper/2406.04178/) (2024)
- [Conv-INR: Convolutional Implicit Neural Representation for Multimodal Visual Signals](https://mlsanity.com/paper/2406.04249/) (2024)
- [Towards the Spectral bias Alleviation by Normalizations in Coordinate Networks](https://mlsanity.com/paper/2407.17834/) (2024)
- [Exploring Kernel Transformations for Implicit Neural Representations](https://mlsanity.com/paper/2504.04728/) (2025)
- [High-Frequency First: A Two-Stage Approach for Improving Image INR](https://mlsanity.com/paper/2508.15582/) (2025)

## Abstract (arXiv)

In this paper, a novel multi-head multi-layer perceptron (MLP) structure is presented for implicit neural representation (INR). Since conventional rectified linear unit (ReLU) networks are shown to exhibit spectral bias towards learning low-frequency features of the signal, we aim at mitigating this defect by taking advantage of the local structure of the signals. To be more specific, an MLP is used to capture the global features of the underlying generator function of the desired signal. Then, several heads are utilized to reconstruct disjoint local features of the signal, and to reduce the computational complexity, sparse layers are deployed for attaching heads to the body. Through various experiments, we show that the proposed model does not suffer from the special bias of conventional ReLU networks and has superior generalization capabilities. Finally, simulation results confirm that the proposed multi-head structure outperforms existing INR methods with considerably less computational cost.