# Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs

arXiv: https://arxiv.org/abs/2111.15135 · HTML: https://mlsanity.com/paper/2111.15135/

## Key facts

- Cited by 135 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 26 of 26)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Texture Fields: Learning Texture Representations in Function Space](https://mlsanity.com/paper/1905.07259/) (2019)
- [DeRF: Decomposed Radiance Fields](https://mlsanity.com/paper/2011.12490/) (2020)
- [NASA: Neural Articulated Shape Approximation](https://mlsanity.com/paper/1912.03207/) (2019)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing](https://mlsanity.com/paper/2108.08807/) (2021)
- [NeuTex: Neural Texture Mapping for Volumetric Neural Rendering](https://mlsanity.com/paper/2103.00762/) (2021)
- [Learning a Neural 3D Texture Space from 2D Exemplars](https://mlsanity.com/paper/1912.04158/) (2019)
- [Rethinking Positional Encoding](https://mlsanity.com/paper/2107.02561/) (2021)
- [CoIL: Coordinate-based Internal Learning for Imaging Inverse Problems](https://mlsanity.com/paper/2102.05181/) (2021)
- [Comparison of non-linear activation functions for deep neural networks on MNIST classification task](https://mlsanity.com/paper/1804.02763/) (2018)
- [Reproducing Activation Function for Deep Learning](https://mlsanity.com/paper/2101.04844/) (2021)
- [LightSAL: Lightweight Sign Agnostic Learning for Implicit Surface Representation](https://mlsanity.com/paper/2103.14273/) (2021)

## Built on by (most cited first, 30 of 135)

- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)](https://mlsanity.com/paper/2210.00379/) (2022)
- [NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors](https://mlsanity.com/paper/2206.13597/) (2022)
- [StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views](https://mlsanity.com/paper/2306.04988/) (2023)
- [F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories](https://mlsanity.com/paper/2303.15951/) (2023)
- [GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting](https://mlsanity.com/paper/2403.08551/) (2024)
- [GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation](https://mlsanity.com/paper/2204.05735/) (2022)
- [COIN++: Neural Compression Across Modalities](https://mlsanity.com/paper/2201.12904/) (2022)
- [FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions](https://mlsanity.com/paper/2312.02434/) (2023)
- [NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions](https://mlsanity.com/paper/2309.15426/) (2023)
- [Spatial Implicit Neural Representations for Global-Scale Species Mapping](https://mlsanity.com/paper/2306.02564/) (2023)
- [FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos](https://mlsanity.com/paper/2212.12294/) (2022)
- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey](https://mlsanity.com/paper/2411.03688/) (2024)
- [Improved Implicit Neural Representation with Fourier Reparameterized Training](https://mlsanity.com/paper/2401.07402/) (2024)
- [INCODE: Implicit Neural Conditioning with Prior Knowledge Embeddings](https://mlsanity.com/paper/2310.18846/) (2023)
- [GECCO: Geometrically-Conditioned Point Diffusion Models](https://mlsanity.com/paper/2303.05916/) (2023)
- [Garment Recovery with Shape and Deformation Priors](https://mlsanity.com/paper/2311.10356/) (2023)
- [A Sampling Theory Perspective on Activations for Implicit Neural Representations](https://mlsanity.com/paper/2402.05427/) (2024)
- [Efficient Learning With Sine-Activated Low-rank Matrices](https://mlsanity.com/paper/2403.19243/) (2024)
- [Intrinsic Neural Fields: Learning Functions on Manifolds](https://mlsanity.com/paper/2203.07967/) (2022)
- [On Quantizing Implicit Neural Representations](https://mlsanity.com/paper/2209.01019/) (2022)
- [Polynomial Implicit Neural Representations For Large Diverse Datasets](https://mlsanity.com/paper/2303.11424/) (2023)
- [HOSC: A Periodic Activation Function for Preserving Sharp Features in Implicit Neural Representations](https://mlsanity.com/paper/2401.10967/) (2024)
- [FreSh: Frequency Shifting for Accelerated Neural Representation Learning](https://mlsanity.com/paper/2410.05050/) (2024)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes](https://mlsanity.com/paper/2406.17438/) (2024)
- [From Activation to Initialization: Scaling Insights for Optimizing Neural Fields](https://mlsanity.com/paper/2403.19205/) (2024)
- [FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representation](https://mlsanity.com/paper/2407.19434/) (2024)

## Abstract (arXiv)

Coordinate-MLPs are emerging as an effective tool for modeling multidimensional continuous signals, overcoming many drawbacks associated with discrete grid-based approximations. However, coordinate-MLPs with ReLU activations, in their rudimentary form, demonstrate poor performance in representing signals with high fidelity, promoting the need for positional embedding layers. Recently, Sitzmann et al. proposed a sinusoidal activation function that has the capacity to omit positional embedding from coordinate-MLPs while still preserving high signal fidelity. Despite its potential, ReLUs are still dominating the space of coordinate-MLPs; we speculate that this is due to the hyper-sensitivity of networks -- that employ such sinusoidal activations -- to the initialization schemes. In this paper, we attempt to broaden the current understanding of the effect of activations in coordinate-MLPs, and show that there exists a broader class of activations that are suitable for encoding signals. We affirm that sinusoidal activations are only a single example in this class, and propose several non-periodic functions that empirically demonstrate more robust performance against random initializations than sinusoids. Finally, we advocate for a shift towards coordinate-MLPs that employ these non-traditional activation functions due to their high performance and simplicity.