mlsanityData to 14 Oct 2026

FNet: Mixing Tokens with Fourier Transforms

arXiv 2105.03824 ↗First posted 9 May 2021cs.CL cs.LGcited by 347

Not on the map. Why?

Abstract (arXiv)

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models outperform Transformer counterparts.

Key facts. Cited by 347 papers in the map (data through 14 Oct 2026). Builds on 52 of its 78 references that are in the map. First posted on arXiv 9 May 2021.

Builds on

52 of 78 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2020Reformer: The Efficient Transformercited by 1,668
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2020Rethinking Attention with Performerscited by 1,601
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  12. 2020Efficient Transformers: A Surveycited by 790
  13. 2018Image Transformercited by 772
  14. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  15. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  16. 2019Designing and Interpreting Probes with Control Taskscited by 640
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  18. 2021Random Feature Attentioncited by 354
  19. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  20. 2020Sparse Sinkhorn Attentioncited by 251
  21. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  22. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  23. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  24. 2020Fast Transformers with Clustered Attentioncited by 123
  25. 2013Fast Training of Convolutional Networks through FFTscited by 89
  26. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  27. 2020Pre-trained Summarization Distillationcited by 71
  28. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  29. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 43
  30. 2015Structured Transforms for Small-Footprint Deep Learningcited by 42
  31. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  32. 2020Language Through a Prism: A Spectral Approach for Multiscale Language Representationscited by 39
  33. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  34. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  35. 2015ACDC: A Structured Efficient Linear Layercited by 19
  36. 2018Learning Long Term Dependencies via Fourier Recurrent Unitscited by 11
  37. 2016Very Efficient Training of Convolutional Neural Networks using Fast Fourier Transform and Overlap-and-Addcited by 7
  38. 2017FFT-Based Deep Learning Deployment in Embedded Systemscited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 36; 2022: 68; 2023: 67; 2024: 69; 2025: 53; 2026: 54.

347 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  2. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  4. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  5. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  7. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 346
  8. 2021Choose a Transformer: Fourier or Galerkincited by 343
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  10. 2022MAXIM: Multi-Axis MLP for Image Processingcited by 302
  11. 2021Global Filter Networks for Image Classificationcited by 260
  12. 2021Factorized Fourier Neural Operatorscited by 249
38 more
  1. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 248
  2. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  3. 2023FITS: Modeling Time Series with 10k Parameterscited by 195
  4. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 148
  5. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  6. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  7. 2022Liquid Structural State-Space Modelscited by 136
  8. 2022MetaFormer Baselines for Visioncited by 132
  9. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  10. 2024Transformers are Multi-State RNNscited by 120
  11. 2021Large-Scale Differentially Private BERTcited by 118
  12. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  13. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  14. 2021The Efficiency Misnomercited by 98
  15. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 82
  16. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  17. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  18. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  19. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  20. 2022Video Transformers: A Surveycited by 63
  21. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  22. 2023MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal and Channel Mixingcited by 53
  23. 2023Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolutioncited by 52
  24. 2022Pretraining Without Attentioncited by 51
  25. 2023SpectFormer: Frequency and Attention is what you need in a Vision Transformercited by 48
  26. 2022Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compressioncited by 45
  27. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 44
  28. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  29. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  30. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  31. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  32. 2023Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equationcited by 32
  33. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  34. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  35. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 31
  36. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  37. 2024Visual Fourier Prompt Tuningcited by 30
  38. 2021Multi-Exit Vision Transformer for Dynamic Inferencecited by 28

50 most cited of 347 All 347 (JSON)