mlsanityData to 7 Oct 2026

Rethinking Positional Encoding

arXiv 2107.02561 ↗First posted 6 Jul 2021cs.LG cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

It is well noted that coordinate based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been solely studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice. Codes available at https://github.com/osiriszjq/Rethinking-positional-encoding.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

32 of 40 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2018On the Spectral Bias of Neural Networkscited by 1,321
  12. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
20 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  3. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  4. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  5. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2017Convolutional Sequence to Sequence Learningcited by 550
  11. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  12. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  13. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  14. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  15. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  16. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  17. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  18. 2019Reconstructing continuous distributions of 3D protein structure from cryo-EM imagescited by 56
  19. 2019Encoding word order in complex embeddingscited by 53
  20. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 7; 2023: 5; 2024: 8; 2025: 6; 2026: 4.

36 papers.

Sort:

By area:

  1. 2021Neural Fields in Visual Computing and Beyondcited by 539
  2. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 134
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  5. 2022GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimationcited by 86
  6. 2023NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functionscited by 65
  7. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  8. 2023Polynomial Neural Fields for Subband Decomposition and Manipulationcited by 22
  9. 2022Intrinsic Neural Fields: Learning Functions on Manifoldscited by 20
  10. 2021LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networkscited by 19
  11. 2024A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationshipscited by 11
  12. 2021An Empirical Study: Extensive Deep Temporal Point Processcited by 10
24 more
  1. 2022Track Targets by Dense Spatio-Temporal Position Encodingcited by 8
  2. 2021Learning Positional Embeddings for Coordinate-MLPscited by 7
  3. 2024HR-INR: Continuous Space-Time Video Super-Resolution via Event Cameracited by 6
  4. 2024Attention Beats Linear for Fast Implicit Neural Representation Generationcited by 6
  5. 2022On Regularizing Coordinate-MLPscited by 5
  6. 2022How You Start Matters for Generalizationcited by 5
  7. 2023Robust Point Cloud Processing through Positional Embeddingcited by 5
  8. 2025Data-Efficient Deep Operator Network for Unsteady Flow: A Multi-Fidelity Approach with Physics-Guided Subsamplingcited by 5
  9. 2022Learning Continuous Implicit Representation for Near-Periodic Patternscited by 4
  10. 2023A Unified Framework for Generative Data Augmentation: A Comprehensive Surveycited by 4
  11. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  12. 2024EulerFormer: Sequential User Behavior Modeling with Complex Vector Attentioncited by 3
  13. 2024Model-based learning for multi-antenna multi-frequency location-to-channel mappingcited by 3
  14. 2025Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fieldscited by 2
  15. 2025OpenGT: A Comprehensive Benchmark For Graph Transformerscited by 2
  16. 2025Gradient Descent as a Shrinkage Operator for Spectral Biascited by 1
  17. 2025When and What to Recommend: Joint Modeling of Timing and Content for Active Sequential Recommendationcited by 1
  18. 2025mmWEAVER: Environment-Specific mmWave Signal Synthesis from a Photo and Activity Descriptioncited by 1
  19. 2023IntraSeismic: a coordinate-based learning approach to seismic inversioncited by 0
  20. 2024GridPE: A Grid Cell-Inspired Unified Position Embedding for Arbitrary-Dimensional Spacescited by 0
  21. 2026Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Studycited by 0
  22. 2026The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedycited by 0
  23. 2026Learning Coordinate-based Convolutional Kernels for Continuous SE(3) Equivariant and Efficient Point Cloud Analysiscited by 0
  24. 2026DD-INR: Dynamics-Driven Implicit Neural Representation for Accelerated Whole-Brain Functional MRI Reconstructioncited by 0