mlsanityData to 7 Oct 2026

On Learning the Transformer Kernel

arXiv 2110.08323 ↗First posted 15 Oct 2021cs.LG cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

In this work we introduce KERNELIZED TRANSFORMER, a generic, scalable, data driven framework for learning the kernel function in Transformers. Our framework approximates the Transformer kernel as a dot product between spectral feature maps and learns the kernel by learning the spectral distribution. This not only helps in learning a generic kernel end-to-end, but also reduces the time and space complexity of Transformers from quadratic to linear. We show that KERNELIZED TRANSFORMERS achieve performance comparable to existing efficient Transformer architectures, both in terms of accuracy as well as computational efficiency. Our study also demonstrates that the choice of the kernel has a substantial impact on performance, and kernel learning variants are competitive alternatives to fixed kernel Transformers, both in long as well as short sequence tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 64 of its 102 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

64 of 102 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2013Auto-Encoding Variational Bayescited by 3,980
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2020Efficient Transformers: A Surveycited by 784
  9. 2018Image Transformercited by 770
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  11. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  13. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  14. 2021Random Feature Attentioncited by 350
  15. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  16. 2020Sparse Sinkhorn Attentioncited by 250
  17. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  18. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  19. 2013Gaussian Process Kernels for Pattern Discovery and Extrapolationcited by 131
  20. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  21. 2020ProGen: Language Modeling for Protein Generationcited by 127
  22. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
  23. 2021An Introduction to Deep Generative Modelingcited by 104
  24. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  25. 2021Luna: Linear Unified Nested Attentioncited by 96
  26. 2016Orthogonal Random Featurescited by 89
  27. 2015On the Error of Random Fourier Featurescited by 76
  28. 2018On GANs and GMMscited by 62
  29. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  30. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  31. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  32. 2017The Unreasonable Effectiveness of Structured Random Orthogonal Embeddingscited by 43
  33. 2014Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernelscited by 41
  34. 2017Non-Stationary Spectral Kernelscited by 38
  35. 2019BP-Transformer: Modelling Long-Range Context via Binary Partitioningcited by 38
  36. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  37. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
  38. 2018But How Does It Work in Theory? Linear SVM with Random Featurescited by 28

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 4; 2025: 0; 2026: 2.

14 papers.

Sort:

  1. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  2. 2023Efficient Attention via Control Variatescited by 23
  3. 2022Chefs' Random Tables: Non-Trigonometric Random Featurescited by 20
  4. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  5. 2023Simplex Random Featurescited by 9
  6. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  7. 2024Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformerscited by 6
  8. 2024Spectraformer: A Unified Random Feature Framework for Transformercited by 3
  9. 2024S3Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketchingcited by 2
  10. 2021SimpleTRON: Simple Transformer with O(N) Complexitycited by 1
  11. 2022Linear Self-Attention Approximation via Trainable Feedforward Kernelcited by 0
  12. 2024EUGens: Efficient, Unified, and General Dense Layerscited by 0
2 more
  1. 2026EUGens: Efficient, Unified, and General Dense Layerscited by 0
  2. 2026From Self-Attention to Connection Laplacian: A Unified Operator View of Transformerscited by 0