mlsanityData to 7 Oct 2026

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Method

arXiv 2111.00035 ↗First posted 29 Oct 2021cs.LG cs.AI cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Transformers are expensive to train due to the quadratic time and space complexity in the self-attention mechanism. On the other hand, although kernel machines suffer from the same computation bottleneck in pairwise dot products, several approximation schemes have been successfully incorporated to considerably reduce their computational cost without sacrificing too much accuracy. In this work, we leverage the computation methods for kernel machines to alleviate the high computational cost and introduce Skyformer, which replaces the softmax structure with a Gaussian kernel to stabilize the model training and adapts the Nystr"om method to a non-positive semidefinite matrix to accelerate the computation. We further conduct theoretical analysis by showing that the matrix approximation error of our proposed method is small in the spectral norm. Experiments on Long Range Arena benchmark show that the proposed method is sufficient in getting comparable or even better performance than the full self-attention while requiring fewer computation resources.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

35 of 51 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
23 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  8. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  9. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  11. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  12. 2020Understanding the Difficulty of Training Transformerscited by 210
  13. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  14. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  15. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  16. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  17. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  18. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  19. 2013Revisiting the Nystrom Method for Improved Large-Scale Machine Learningcited by 76
  20. 2015Randomized sketches for kernels: Fast and optimal non-parametric regressioncited by 31
  21. 2018Hashing-Based-Estimators for Kernel Density in High Dimensionscited by 31
  22. 2021Fast Statistical Leverage Score Approximation in Kernel Ridge Regressioncited by 14
  23. 2021Accumulation of Sub-Sampling Matrices with Applications to Statistical Computationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 11; 2024: 12; 2025: 18; 2026: 7.

61 papers.

Sort:

By area:

  1. 2022On The Computational Complexity of Self-Attentioncited by 137
  2. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  3. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  4. 2022The Devil in Linear Transformercited by 98
  5. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  6. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  7. 2023KDEformer: Accelerating Transformers via Kernel Density Estimationcited by 58
  8. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  9. 2023In-Context Learning with Many Demonstration Examplescited by 39
  10. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  11. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  12. 2025PolaFormer: Polarity-aware Linear Attention for Vision Transformerscited by 30
38 more
  1. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  2. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  3. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  4. 2024Encryption-Friendly LLM Architecturecited by 15
  5. 2024Towards LifeSpan Cognitive Systemscited by 13
  6. 2023Linear Log-Normal Attention with Unbiased Concentrationcited by 11
  7. 2024Unraveling the Gradient Descent Dynamics of Transformerscited by 11
  8. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10
  9. 2025ZETA: Leveraging Z-order Curves for Efficient Top-k Attentioncited by 9
  10. 2023Exploring Transformer Extrapolationcited by 8
  11. 2025Revisiting associative recall in modern recurrent modelscited by 7
  12. 2022Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attentioncited by 6
  13. 2026WildCat: Near-Linear Attention in Theory and Practicecited by 6
  14. 2021Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequencescited by 5
  15. 2022Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networkscited by 5
  16. 2022Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layerscited by 5
  17. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  18. 2025Fixed-Point RNNs: Interpolating from Diagonal to Densecited by 5
  19. 2025Riemannian Optimization on Relaxed Indicator Matrix Manifoldcited by 5
  20. 2024Expanding Expressivity in Transformer Models with M"obiusAttentioncited by 4
  21. 2025LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transportcited by 4
  22. 2022Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetriescited by 3
  23. 2022Inducer-tuning: Connecting Prefix-tuning and Adapter-tuningcited by 3
  24. 2022DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attentioncited by 3
  25. 2023FLuRKA: Fast and accurate unified Low-Rank & Kernel Attentioncited by 3
  26. 2024Spectraformer: A Unified Random Feature Framework for Transformercited by 3
  27. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  28. 2026LaplacianFormer:Rethinking Linear Attention with Laplacian Kernelcited by 3
  29. 2024From Darkness to Detail: Frequency-Aware SSMs for Low-Light Visioncited by 2
  30. 2025A unified framework for establishing the universal approximation of transformer-type architecturescited by 2
  31. 2025SchoenbAt: Rethinking Attention with Polynomial basiscited by 1
  32. 2025A Framework for Non-Linear Attention via Modern Hopfield Networkscited by 1
  33. 2023TAP: The Attention Patch for Cross-Modal Knowledge Transfer from Unlabeled Modalitycited by 0
  34. 2023SKI to go Faster: Accelerating Toeplitz Neural Networks via Asymmetric Kernelscited by 0
  35. 2023Manifold-Preserving Transformers are Effective for Short-Long Range Encodingcited by 0
  36. 2024Kernel Approximation using Analog In-Memory Computingcited by 0
  37. 2025Irrational Complex Rotations Empower Low-bit Optimizerscited by 0
  38. 2025Converting Transformers into DGNNs Formcited by 0

50 most cited of 61 All 61 (JSON)