mlsanityData to 14 Oct 2026

Elliptical Attention

arXiv 2406.13770 ↗First posted 19 Jun 2024cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. This dot-product self-attention computes attention weights among the input tokens using Euclidean distance, which makes the model prone to representation collapse and vulnerable to contaminated samples. In this paper, we propose using a Mahalanobis distance metric for computing the attention weights to stretch the underlying feature space in directions of high contextual relevance. In particular, we define a hyper-ellipsoidal neighborhood around each query to increase the attention weights of the tokens lying in the contextually important directions. We term this novel class of attention Elliptical Attention. Our Elliptical Attention provides two benefits: 1) reducing representation collapse and 2) enhancing the model's robustness as Elliptical Attention pays more attention to contextually relevant information rather than focusing on some small subset of informative features. We empirically demonstrate the advantages of Elliptical Attention over the baseline dot-product attention and state-of-the-art attention methods on various practical tasks, including object classification, image segmentation, and language modeling across different data modalities.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 63 of its 90 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

63 of 90 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  10. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  13. 2021Transformers in Vision: A Surveycited by 1,013
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  15. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  16. 2020Efficient Transformers: A Surveycited by 790
  17. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  18. 2018Universal Transformerscited by 647
  19. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  20. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  21. 2021A Survey of Transformerscited by 401
  22. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  23. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  24. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  25. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  26. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  27. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 213
  28. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  29. 2022Understanding The Robustness in Vision Transformerscited by 144
  30. 2023The emergence of clusters in self-attention dynamicscited by 140
  31. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  32. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  33. 2021Towards Robust Vision Transformercited by 137
  34. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  35. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  36. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  37. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 91
  38. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Tight Clusters Make Specialized Expertscited by 8
  2. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  3. 2025Robust Filter Attention: Self-Attention as Precision-Weighted State Estimationcited by 1
  4. 2026QUEST: A robust attention formulation using query-modulated spherical attentioncited by 1
  5. 2026Invertible Query-Key Coupling Composes with Attention Mechanismscited by 1
  6. 2025Revisiting Transformers with Insights from Image Filtering and Boostingcited by 0
  7. 2026Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noisecited by 0