mlsanityData to 7 Oct 2026

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

arXiv 2302.01925 ↗First posted 3 Feb 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These include regular RPE techniques applied for sequential data, as well as novel RPEs operating on geometric data embedded in higher-dimensional Euclidean spaces. FLTs construct the optimal RPE mechanism implicitly by learning its spectral representation. As opposed to other architectures combining efficient low-rank linear attention with RPEs, FLTs remain practical in terms of their memory usage and do not require additional assumptions about the structure of the RPE mask. Besides, FLTs allow for applying certain structural inductive bias techniques to specify masking strategies, e.g. they provide a way to learn the so-called local RPEs introduced in this paper and give accuracy gains as compared with several other linear Transformers for language modeling. We also thoroughly test FLTs on other data modalities and tasks, such as image classification, 3D molecular modeling, and learnable optimizers. To the best of our knowledge, for 3D molecular data, FLTs are the first Transformer architectures providing linear attention and incorporating RPE masking.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 61 of its 90 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

61 of 90 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2018Self-Attention with Relative Position Representationscited by 1,112
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2022A Generalist Agentcited by 802
  13. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  14. 2018Universal Transformerscited by 613
  15. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  16. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  17. 2021Random Feature Attentioncited by 350
  18. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  19. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  20. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  21. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  22. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  23. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  24. 2020Fast Transformers with Clustered Attentioncited by 123
  25. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  26. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  27. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  28. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  29. 2018Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guaranteescited by 71
  30. 2022Benchmarking Graphormer on Large-Scale Molecular Modeling Datasetscited by 63
  31. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  32. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  33. 2021What Context Features Can Transformer Language Models Use?cited by 49
  34. 2020Language Through a Prism: A Spectral Approach for Multiscale Language Representationscited by 39
  35. 2021From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformerscited by 38
  36. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 36
  37. 2022Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigationcited by 27
  38. 2021Hybrid Random Featurescited by 25

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 1; 2026: 3.

10 papers.

  1. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  2. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  3. 2024Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physicscited by 12
  4. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  5. 2024Structure-informed Positional Encoding for Music Generationcited by 5
  6. 2025Beyond Structure: Invariant Crystal Property Prediction with Pseudo-Particle Ray Diffractioncited by 4
  7. 2024SeTformer is What You Need for Vision and Languagecited by 3
  8. 2026You Need Better Attention Priorscited by 2
  9. 2026SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernelcited by 1
  10. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0