# Trading Positional Complexity vs. Deepness in Coordinate Networks

arXiv: https://arxiv.org/abs/2205.08987 · HTML: https://mlsanity.com/paper/2205.08987/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 18 May 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video](https://mlsanity.com/paper/1702.00824/) (2017)
- [Learning to Encode Position for Transformer with Continuous Dynamical Model](https://mlsanity.com/paper/2003.09229/) (2020)

## Built on by (most cited first, 16 of 16)

- [Spatial Implicit Neural Representations for Global-Scale Species Mapping](https://mlsanity.com/paper/2306.02564/) (2023)
- [Fast Neural Scene Flow](https://mlsanity.com/paper/2304.09121/) (2023)
- [A Sampling Theory Perspective on Activations for Implicit Neural Representations](https://mlsanity.com/paper/2402.05427/) (2024)
- [Floxels: Fast Unsupervised Voxel Based Scene Flow Estimation](https://mlsanity.com/paper/2503.04718/) (2025)
- [Curvature-Aware Training for Coordinate Networks](https://mlsanity.com/paper/2305.08552/) (2023)
- [Robust Point Cloud Processing through Positional Embedding](https://mlsanity.com/paper/2309.00339/) (2023)
- [RBF-PINN: Non-Fourier Positional Embedding in Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.08367/) (2024)
- [Preconditioners for the Stochastic Training of Neural Fields](https://mlsanity.com/paper/2402.08784/) (2024)
- [ARC: Anchored Representation Clouds for High-Resolution INR Classification](https://mlsanity.com/paper/2503.15156/) (2025)
- [Feature Mapping in Physics-Informed Neural Networks (PINNs)](https://mlsanity.com/paper/2402.06955/) (2024)
- [Gaussian Splatting Lucas-Kanade](https://mlsanity.com/paper/2407.11309/) (2024)
- [Diff-INR: Generative Regularization for Electrical Impedance Tomography](https://mlsanity.com/paper/2409.04494/) (2024)
- [CoMI-IRL: Contrastive Multi-Intention Inverse Reinforcement Learning](https://mlsanity.com/paper/2602.07496/) (2026)
- [Fast Kernel Scene Flow](https://mlsanity.com/paper/2403.05896/) (2024)
- [Gradient Descent as a Shrinkage Operator for Spectral Bias](https://mlsanity.com/paper/2504.18207/) (2025)
- [Learning Coordinate-based Convolutional Kernels for Continuous SE(3) Equivariant and Efficient Point Cloud Analysis](https://mlsanity.com/paper/2603.17538/) (2026)

## Abstract (arXiv)

It is well noted that coordinate-based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been mainly studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. In addition, we argue that employing a more complex positional encoding -- that scales exponentially with the number of modes -- requires only a linear (rather than deep) coordinate function to achieve comparable performance. Counter-intuitively, we demonstrate that trading positional embedding complexity for network deepness is orders of magnitude faster than current state-of-the-art; despite the additional embedding complexity. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice.