mlsanityData to 7 Oct 2026

FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention

arXiv 2108.02347 ↗First posted 5 Aug 2021cs.LG cs.AI cs.NAcited by 26

Not on the map. Why?

Abstract (arXiv)

We propose FMMformers, a class of efficient and flexible transformers inspired by the celebrated fast multipole method (FMM) for accelerating interacting particle simulation. FMM decomposes particle-particle interaction into near-field and far-field components and then performs direct and coarse-grained computation, respectively. Similarly, FMMformers decompose the attention into near-field and far-field attention, modeling the near-field attention by a banded matrix and the far-field attention by a low-rank matrix. Computing the attention matrix for FMMformers requires linear complexity in computational time and memory footprint with respect to the sequence length. In contrast, standard transformers suffer from quadratic complexity. We analyze and validate the advantage of FMMformers over the standard transformer on the Long Range Arena and language modeling benchmarks. FMMformers can even outperform the standard transformer in terms of accuracy by a significant margin. For instance, FMMformers achieve an average classification accuracy of 60.74% over the five Long Range Arena tasks, which is significantly better than the standard transformer's average accuracy of 58.70%.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 5 Aug 2021.

Builds on

54 of 76 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
38 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2018Image Transformercited by 770
  15. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  16. 2019Designing and Interpreting Probes with Control Taskscited by 608
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  19. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  20. 2019Axial Attention in Multidimensional Transformerscited by 350
  21. 2021Random Feature Attentioncited by 350
  22. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  23. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  24. 2021Choose a Transformer: Fourier or Galerkincited by 333
  25. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  26. 2020Sparse Sinkhorn Attentioncited by 250
  27. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  28. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  29. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  30. 2018Music Transformercited by 207
  31. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  32. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  33. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  34. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  35. 2020Fast Transformers with Clustered Attentioncited by 123
  36. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  37. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  38. 2017Structured Attention Networkscited by 95

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 1; 2024: 5; 2025: 5; 2026: 3.

26 papers.

Sort:

By area:

  1. 2022Mitigating spectral bias for the multiscale operator learningcited by 43
  2. 2025Log-Linear Attentioncited by 39
  3. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  4. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  5. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  6. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  7. 2023Efficient Attention via Control Variatescited by 23
  8. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  9. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 18
  10. 2021On Learning the Transformer Kernelcited by 14
  11. 2022DCT-Former: Efficient Self-Attention with Discrete Cosine Transformcited by 12
  12. 2026Memory Caching: RNNs with Growing Memorycited by 12
14 more
  1. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  2. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  3. 2022Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layerscited by 5
  4. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequencescited by 5
  5. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5
  6. 2022Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearizationcited by 4
  7. 2022DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attentioncited by 3
  8. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  9. 2024S3Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketchingcited by 2
  10. 2025Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision Transformers?cited by 2
  11. 2024ATP: Enabling Fast LLM Serving via Attention on Top Principal Keyscited by 1
  12. 2025Transformer Based Linear Attention with Optimized GPU Kernel Implementationcited by 1
  13. 2026DnA: Denoising Attention for Visual Taskscited by 0
  14. 2026EMAGN: Efficient Multi-Attention Graph Network via Learned Clustering for Scalable Traffic Forecastingcited by 0