Fast Monte-Carlo Approximation of the Attention Mechanism
Not on the map. Why?
Abstract (arXiv)
We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA exploits the fact that the importance of each token in an input sequence varies with respect to their attention scores; thus, some degree of error can be tolerable when encoding tokens with low attention. Using approximate matrix multiplication, MCA applies different error bounds to encode input tokens such that those with low attention scores are computed with relaxed precision, whereas errors of salient elements are minimized. MCA can operate in parallel with other attention optimization schemes and does not require model modification. We study the theoretical error bounds and demonstrate that MCA reduces attention complexity (in FLOPS) for various Transformer models by up to 11× in GLUE benchmarks without compromising model accuracy.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 30 Jan 2022.
Builds on
30 of 43 references.
References · 30
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2020Reformer: The Efficient Transformercited by 1,658
- 2020Rethinking Attention with Performerscited by 1,584
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
18 more
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2020Efficient Transformers: A Surveycited by 784
- 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
- 2019Stand-Alone Self-Attention in Vision Modelscited by 542
- 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
- 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
- 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
- 2019Q8BERT: Quantized 8Bit BERTcited by 296
- 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
- 2019Structured Pruning of Large Language Modelscited by 201
- 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
- 2019Blockwise Self-Attention for Long Document Understandingcited by 158
- 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
- 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
- 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
- 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
- 2019NAT: Neural Architecture Transformer for Accurate and Compact Architecturescited by 33
- 2019Vector Quantized Bayesian Neural Network Inference for Data Streamscited by 5
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 2.
3 papers.