Positional Embedding-Aware Activations
Not on the map. Why?
Abstract (arXiv)
We present a neural network architecture designed to naturally learn a positional embedding and overcome the spectral bias towards lower frequencies faced by conventional activation functions. Our proposed architecture, SPDER, is a simple MLP that uses an activation function composed of a sinusoidal multiplied by a sublinear function, called the damping function. The sinusoidal enables the network to automatically learn the positional embedding of an input coordinate while the damping passes on the actual coordinate value by preventing it from being projected down to within a finite range of values. Our results indicate that SPDERs speed up training by 10x and converge to losses 1,500-50,000x lower than that of the state-of-the-art for image representation. SPDER is also state-of-the-art in audio representation. The superior representation capability allows SPDER to also excel on multiple downstream tasks such as image super-resolution and video frame interpolation. We provide intuition as to why SPDER significantly improves fitting compared to that of other INR methods while requiring no hyperparameter tuning or preprocessing.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 27 Jun 2023.
Builds on
25 of 36 references.
References · 25
- 2017Attention Is All You Needcited by 60,361
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
- 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
- 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
- 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
- 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
- 2018On the Spectral Bias of Neural Networkscited by 1,341
- 2016End-to-end Optimized Image Compressioncited by 696
- 2021NeRV: Neural Representations for Videoscited by 256
13 more
- 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
- 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
- 2018Training behavior of deep neural network in frequency domaincited by 174
- 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
- 2021Implicit Neural Representations for Image Compressioncited by 162
- 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 135
- 2022VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolutioncited by 101
- 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
- 2023HNeRV: A Hybrid Neural Representation for Videoscited by 100
- 2022An Introduction to Neural Data Compressioncited by 79
- 2017Learning Convolutional Networks for Content-weighted Image Compressioncited by 76
- 2022Your Transformer May Not be as Powerful as You Expectcited by 55
- 2022Optical Flow Regularization of Implicit Neural Representations for Video Frame Interpolationcited by 1
Built on by
Citing papers per year: 2024: 2.
2 papers.