# DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

arXiv: https://arxiv.org/abs/1709.04696 · HTML: https://mlsanity.com/paper/1709.04696/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2017.

## Built on by (most cited first, 30 of 71)

- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [SAITS: Self-Attention-based Imputation for Time Series](https://mlsanity.com/paper/2202.08516/) (2022)
- [Self-Attention Networks Can Process Bounded Hierarchical Languages](https://mlsanity.com/paper/2105.11115/) (2021)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Boosting Crowd Counting via Multifaceted Attention](https://mlsanity.com/paper/2203.02636/) (2022)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2109.01876/) (2021)
- [Fine-Grained Distillation for Long Document Retrieval](https://mlsanity.com/paper/2212.10423/) (2022)
- [Semi-supervised Credit Card Fraud Detection via Attribute-Driven Graph Representation](https://mlsanity.com/paper/2412.18287/) (2024)
- [QSAN: A Near-term Achievable Quantum Self-Attention Network](https://mlsanity.com/paper/2207.07563/) (2022)
- [Focus U-Net: A novel dual attention-gated CNN for polyp segmentation during colonoscopy](https://mlsanity.com/paper/2105.07467/) (2021)
- [Modality specific U-Net variants for biomedical image segmentation: A survey](https://mlsanity.com/paper/2107.04537/) (2021)
- [A Survey on Aspect-Based Sentiment Classification](https://mlsanity.com/paper/2203.14266/) (2022)
- [LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching](https://mlsanity.com/paper/2102.12671/) (2021)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)
- [Simulating Hard Attention Using Soft Attention](https://mlsanity.com/paper/2412.09925/) (2024)
- [Paraphrase Identification with Deep Learning: A Review of Datasets and Methods](https://mlsanity.com/paper/2212.06933/) (2022)
- [Linguistically-Enriched and Context-Aware Zero-shot Slot Filling](https://mlsanity.com/paper/2101.06514/) (2021)
- [Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation](https://mlsanity.com/paper/2103.00819/) (2021)
- [AWSnet: An Auto-weighted Supervision Attention Network for Myocardial Scar and Edema Segmentation in Multi-sequence Cardiac Magnetic Resonance Images](https://mlsanity.com/paper/2201.05344/) (2022)
- [LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding](https://mlsanity.com/paper/2505.16983/) (2025)
- [AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding](https://mlsanity.com/paper/2507.02591/) (2025)
- [Neural Attention Models in Deep Learning: Survey and Taxonomy](https://mlsanity.com/paper/2112.05909/) (2021)
- [TENT: Tensorized Encoder Transformer for Temperature Forecasting](https://mlsanity.com/paper/2106.14742/) (2021)
- [AMD-HookNet for Glacier Front Segmentation](https://mlsanity.com/paper/2302.02744/) (2023)
- [A Simple Meta-learning Paradigm for Zero-shot Intent Classification with Mixture Attention Mechanism](https://mlsanity.com/paper/2206.02179/) (2022)
- [State of the Art and Potentialities of Graph-level Learning](https://mlsanity.com/paper/2301.05860/) (2023)
- [PAT: Position-Aware Transformer for Dense Multi-Label Action Detection](https://mlsanity.com/paper/2308.05051/) (2023)

## Abstract (arXiv)

Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02% on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets.