mlsanityData to 7 Oct 2026

Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding

arXiv 2106.02795 ↗First posted 5 Jun 2021cs.LG cs.AI cs.CVcited by 93

Not on the map. Why?

Abstract (arXiv)

Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e.g., pixel positions on an image, where L2 distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
20 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  3. 2018Image Transformercited by 770
  4. 2017Convolutional Sequence to Sequence Learningcited by 550
  5. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  6. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  7. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
  8. 2019Attention Augmented Convolutional Networkscited by 363
  9. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  10. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  11. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 72
  12. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  13. 2019Encoding word order in complex embeddingscited by 53
  14. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  15. 2014Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernelscited by 41
  16. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 36
  17. 2018But How Does It Work in Theory? Linear SVM with Random Featurescited by 28
  18. 2019Self-Attention with Structural Position Representationscited by 21
  19. 2019Recurrent Space-time Graph Neural Networkscited by 19
  20. 2013Compact Random Feature Mapscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 9; 2024: 20; 2025: 26; 2026: 24.

93 papers.

Sort:

By area:

  1. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  2. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  3. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  4. 2024Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimationcited by 57
  5. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  6. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  7. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  8. 2022Pre-training General Trajectory Embeddings with Maximum Multi-view Entropy Codingcited by 25
  9. 2023Localizing Moments in Long Video Via Multimodal Guidancecited by 23
  10. 2023Unifying Top-down and Bottom-up Scanpath Prediction Using Transformerscited by 22
  11. 2021Can Vision Transformers Perform Convolution?cited by 18
  12. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
38 more
  1. 2022Gaussian Radar Transformer for Semantic Segmentation in Noisy Radar Datacited by 15
  2. 2022Generalized Classification of Satellite Image Time Series with Thermal Positional Encodingcited by 14
  3. 2024Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcriptioncited by 14
  4. 2025CoMatch: Dynamic Covisibility-Aware Transformer for Bilateral Subpixel-Level Semi-Dense Image Matchingcited by 12
  5. 2024Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equationscited by 11
  6. 2022Learnable Spectral Wavelets on Dynamic Graphs to Capture Global Interactionscited by 10
  7. 2023A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Spacecited by 10
  8. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  9. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 10
  10. 2023Radar Instance Transformer: Reliable Moving Instance Segmentation in Sparse Radar Point Cloudscited by 9
  11. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 9
  12. 2025What Happens Next? Anticipating Future Motion by Generating Point Trajectoriescited by 9
  13. 2024Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Datacited by 8
  14. 2022City-scale Incremental Neural Mapping with Three-layer Sampling and Panoptic Representationcited by 7
  15. 2022A unified model for continuous conditional video predictioncited by 7
  16. 2024Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentationscited by 7
  17. 2025VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulationcited by 7
  18. 2023PAT: Position-Aware Transformer for Dense Multi-Label Action Detectioncited by 6
  19. 2024A self-supervised framework for learning whole slide representationscited by 6
  20. 2025Referring Expression Comprehension for Small Objectscited by 6
  21. 2026FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulationcited by 6
  22. 2023STS-CCL: Spatial-Temporal Synchronous Contextual Contrastive Learning for Urban Traffic Forecastingcited by 5
  23. 2025Few-shot Personalized Scanpath Predictioncited by 5
  24. 2025Understanding the Limits of Deep Tabular Methods with Temporal Shiftcited by 4
  25. 2025TransferTraj: A Vehicle Trajectory Learning Model for Region and Task Transferabilitycited by 4
  26. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  27. 2026UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observationcited by 4
  28. 2022PIP: Positional-encoding Image Priorcited by 3
  29. 2024UVTM: Universal Vehicle Trajectory Modeling with ST Feature Domain Generationcited by 3
  30. 2026ST-π: Structured SpatioTemporal VLA for Robotic Manipulationcited by 3
  31. 2022Multi-scale Sinusoidal Embeddings Enable Learning on High Resolution Mass Spectrometry Datacited by 2
  32. 2024Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicinecited by 2
  33. 2024STNAGNN: Data-driven Spatio-temporal Brain Connectivity beyond FCcited by 2
  34. 2024Generative Feature Training of Thin 2-Layer Networkscited by 2
  35. 2025Geography-Aware Large Language Models for Next POI Recommendationcited by 2
  36. 2025DeepTopoNet: A Framework for Subglacial Topography Estimation on the Greenland Ice Sheetscited by 2
  37. 2025Neural Functions for Learning Periodic Signalcited by 2
  38. 2025Score-informed Neural Operator for Enhancing Ordering-based Causal Discoverycited by 2

50 most cited of 93 All 93 (JSON)