mlsanityData to 14 Oct 2026

Exclusive Self Attention

arXiv 2603.09078 ↗First posted 10 Mar 2026cs.LG cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 7 of its 15 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

7 of 15 references.

References · 7

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing preventioncited by 1
  2. 2026Improved Belief-Attention in Vision Taskcited by 0
  3. 2026Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline)cited by 0
  4. 2026NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inferencecited by 0
  5. 2026SAGE: Semantic Audio Generative Encodercited by 0
  6. 2026Looped Diffusion Transformercited by 0