mlsanityData to 7 Oct 2026

DARTFormer: Finding The Best Type Of Attention

arXiv 2210.00641 ↗First posted 2 Oct 2022cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Given the wide and ever growing range of different efficient Transformer attention mechanisms, it is important to identify which attention is most effective when given a task. In this work, we are also interested in combining different attention types to build heterogeneous Transformers. We first propose a DARTS-like Neural Architecture Search (NAS) method to find the best attention for a given task, in this setup, all heads use the same attention (homogeneous models). Our results suggest that NAS is highly effective on this task, and it identifies the best attention mechanisms for IMDb byte level text classification and Listops. We then extend our framework to search for and build Transformers with multiple different attention types, and call them heterogeneous Transformers. We show that whilst these heterogeneous Transformers are better than the average homogeneous models, they cannot outperform the best. We explore the reasons why heterogeneous attention makes sense, and why it ultimately fails.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 2 Oct 2022.

Builds on

23 of 25 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2020Efficient Transformers: A Surveycited by 784
11 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  2. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  3. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  4. 2020Sparse Sinkhorn Attentioncited by 250
  5. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  6. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  7. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  8. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  9. 2022Neural Architecture Search on Efficient Transformers and Beyondcited by 17
  10. 2021RankNAS: Efficient Neural Architecture Search by Pairwise Rankingcited by 12
  11. 2020Finding Fast Transformers: One-Shot Neural Architecture Search by Component Compositioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 0; 2026: 1.

3 papers.

  1. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  2. 2023SlowFormer: Universal Adversarial Patch for Attack on Compute and Energy Efficiency of Inference Efficient Vision Transformerscited by 3
  3. 2026MIDAS: Mosaic Input-Specific Differentiable Architecture Searchcited by 0