# Multi Resolution Analysis (MRA) for Approximate Self-Attention

arXiv: https://arxiv.org/abs/2207.10284 · HTML: https://mlsanity.com/paper/2207.10284/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [Scatterbrain: Unifying Sparse and Low-rank Attention Approximation](https://mlsanity.com/paper/2110.15343/) (2021)
- [SOFT: Softmax-free Transformer with Linear Complexity](https://mlsanity.com/paper/2110.11945/) (2021)
- [H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences](https://mlsanity.com/paper/2107.11906/) (2021)
- [You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling](https://mlsanity.com/paper/2111.09714/) (2021)
- [Connections Between Nuclear Norm and Frobenius Norm Based Representations](https://mlsanity.com/paper/1502.07423/) (2015)
- [The Incremental Multiresolution Matrix Factorization Algorithm](https://mlsanity.com/paper/1705.05804/) (2017)

## Built on by (most cited first, 11 of 11)

- [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025)
- [Log-Linear Attention](https://mlsanity.com/paper/2506.04761/) (2025)
- [Efficient Attention via Control Variates](https://mlsanity.com/paper/2302.04542/) (2023)
- [Memory Caching: RNNs with Growing Memory](https://mlsanity.com/paper/2602.24281/) (2026)
- [LookupFFN: Making Transformers Compute-lite for CPU inference](https://mlsanity.com/paper/2403.07221/) (2024)
- [Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens](https://mlsanity.com/paper/2305.04241/) (2023)
- [Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers](https://mlsanity.com/paper/2512.16615/) (2025)
- [Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics](https://mlsanity.com/paper/2507.02748/) (2025)
- [Fast Multipole Attention: A Scalable Multilevel Attention Mechanism for Text and Images](https://mlsanity.com/paper/2310.11960/) (2023)
- [Superlinear Multi-Step Attention](https://mlsanity.com/paper/2601.18401/) (2026)
- [CoFusion: Multispectral and Hyperspectral Image Fusion via Spectral Coordinate Attention](https://mlsanity.com/paper/2604.10584/) (2026)

## Abstract (arXiv)

Transformers have emerged as a preferred model for many tasks in natural langugage processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at urlhttps://github.com/mlpen/mra-attention.