# How Does Selective Mechanism Improve Self-Attention Networks?

arXiv: https://arxiv.org/abs/2005.00979 · HTML: https://mlsanity.com/paper/2005.00979/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2020.

## Built on by (most cited first, 19 of 19)

- [MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts](https://mlsanity.com/paper/2407.21770/) (2024)
- [Improving Visual Prompt Tuning for Self-supervised Vision Transformers](https://mlsanity.com/paper/2306.05067/) (2023)
- [Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation](https://mlsanity.com/paper/2403.11808/) (2024)
- [Gumbel-Attention for Multi-modal Machine Translation](https://mlsanity.com/paper/2103.08862/) (2021)
- [Towards Trustworthy Explanation: On Causal Rationalization](https://mlsanity.com/paper/2306.14115/) (2023)
- [Neural News Recommendation with Collaborative News Encoding and Structural User Encoding](https://mlsanity.com/paper/2109.00750/) (2021)
- [Not All Attention Is All You Need](https://mlsanity.com/paper/2104.04692/) (2021)
- [Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling](https://mlsanity.com/paper/2106.10840/) (2021)
- [Global memory transformer for processing long documents](https://mlsanity.com/paper/2212.01650/) (2022)
- [Learning to Select Prototypical Parts for Interpretable Sequential Data Modeling](https://mlsanity.com/paper/2212.03396/) (2022)
- [Improving Speech Emotion Recognition Through Focus and Calibration Attention Mechanisms](https://mlsanity.com/paper/2208.10491/) (2022)
- [AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions](https://mlsanity.com/paper/2509.01787/) (2025)
- [Drop Edges and Adapt: a Fairness Enforcing Fine-tuning for Graph Neural Networks](https://mlsanity.com/paper/2302.11479/) (2023)
- [Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Space](https://mlsanity.com/paper/2307.13995/) (2023)
- [A Learnable Counter-condition Analysis Framework for Functional Connectivity-based Neurological Disorder Diagnosis](https://mlsanity.com/paper/2310.03964/) (2023)
- [Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability](https://mlsanity.com/paper/2505.03097/) (2025)
- [Text Rationalization for Robust Causal Effect Estimation](https://mlsanity.com/paper/2512.05373/) (2025)
- [Enhancing Job Salary Prediction with Disentangled Composition Effect Modeling: A Neural Prototyping Approach](https://mlsanity.com/paper/2503.12978/) (2025)
- [Bayesian Gated Non-Negative Contrastive Learning](https://mlsanity.com/paper/2605.28441/) (2026)

## Abstract (arXiv)

Self-attention networks (SANs) with selective mechanism has produced substantial improvements in various NLP tasks by concentrating on a subset of input words. However, the underlying reasons for their strong performance have not been well explained. In this paper, we bridge the gap by assessing the strengths of selective SANs (SSANs), which are implemented with a flexible and universal Gumbel-Softmax. Experimental results on several representative NLP tasks, including natural language inference, semantic role labelling, and machine translation, show that SSANs consistently outperform the standard SANs. Through well-designed probing experiments, we empirically validate that the improvement of SSANs can be attributed in part to mitigating two commonly-cited weaknesses of SANs: word order encoding and structure modeling. Specifically, the selective mechanism improves SANs by paying more attention to content words that contribute to the meaning of the sentence. The code and data are released at https://github.com/xwgeng/SSAN.