# Distance-based Self-Attention Network for Natural Language Inference

arXiv: https://arxiv.org/abs/1712.02047 · HTML: https://mlsanity.com/paper/1712.02047/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2017.

## Built on by (most cited first, 6 of 6)

- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [Comateformer: Combined Attention Transformer for Semantic Sentence Matching](https://mlsanity.com/paper/2412.07220/) (2024)
- [CellularLint: A Systematic Approach to Identify Inconsistent Behavior in Cellular Network Specifications](https://mlsanity.com/paper/2407.13742/) (2024)
- [LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching](https://mlsanity.com/paper/2108.02915/) (2021)
- [DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks](https://mlsanity.com/paper/2204.08688/) (2022)
- [DGA-Net Dynamic Gaussian Attention Network for Sentence Semantic Matching](https://mlsanity.com/paper/2106.04905/) (2021)

## Abstract (arXiv)

Attention mechanism has been used as an ancillary means to help RNN or CNN. However, the Transformer (Vaswani et al., 2017) recently recorded the state-of-the-art performance in machine translation with a dramatic reduction in training time by solely using attention. Motivated by the Transformer, Directional Self Attention Network (Shen et al., 2017), a fully attention-based sentence encoder, was proposed. It showed good performance with various data by using forward and backward directional information in a sentence. But in their study, not considered at all was the distance between words, an important feature when learning the local dependency to help understand the context of input text. We propose Distance-based Self-Attention Network, which considers the word distance by using a simple distance mask in order to model the local dependency without losing the ability of modeling global dependency which attention has inherent. Our model shows good performance with NLI data, and it records the new state-of-the-art result with SNLI data. Additionally, we show that our model has a strength in long sentences or documents.