# Convolutional Self-Attention Networks

arXiv: https://arxiv.org/abs/1904.03107 · HTML: https://mlsanity.com/paper/1904.03107/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 30 of 32)

- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Mask Attention Networks: Rethinking and Strengthen Transformer](https://mlsanity.com/paper/2103.13597/) (2021)
- [Anomaly Detection in Dynamic Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2406.00134/) (2024)
- [Graph-Enhanced Multi-Task Learning of Multi-Level Transition Dynamics for Session-based Recommendation](https://mlsanity.com/paper/2110.03996/) (2021)
- [Multi-head or Single-head? An Empirical Comparison for Transformer Training](https://mlsanity.com/paper/2106.09650/) (2021)
- [The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT](https://mlsanity.com/paper/2101.09115/) (2021)
- [An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2106.09885/) (2021)
- [Non-equispaced Fourier Neural Solvers for PDEs](https://mlsanity.com/paper/2212.04689/) (2022)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation](https://mlsanity.com/paper/2310.09998/) (2023)
- [Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding](https://mlsanity.com/paper/2204.02922/) (2022)
- [SkipConvGAN: Monaural Speech Dereverberation using Generative Adversarial Networks via Complex Time-Frequency Masking](https://mlsanity.com/paper/2211.12623/) (2022)
- [A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition](https://mlsanity.com/paper/2304.07611/) (2023)
- [BERT-based Ensemble Approaches for Hate Speech Detection](https://mlsanity.com/paper/2209.06505/) (2022)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [An Attention Module for Convolutional Neural Networks](https://mlsanity.com/paper/2108.08205/) (2021)
- [Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers](https://mlsanity.com/paper/2110.02402/) (2021)
- [Attention-based Neural Beamforming Layers for Multi-channel Speech Recognition](https://mlsanity.com/paper/2105.05920/) (2021)
- [Extreme precipitation forecasting using attention augmented convolutions](https://mlsanity.com/paper/2201.13408/) (2022)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translation](https://mlsanity.com/paper/2312.15872/) (2023)
- [To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph](https://mlsanity.com/paper/2101.06397/) (2021)
- [Cascaded Head-colliding Attention](https://mlsanity.com/paper/2105.14850/) (2021)
- [Hero-Gang Neural Model For Named Entity Recognition](https://mlsanity.com/paper/2205.07177/) (2022)
- [URM4DMU: an user represention model for darknet markets users](https://mlsanity.com/paper/2303.10674/) (2023)
- [Self-Attention-Based Contextual Modulation Improves Neural System Identification](https://mlsanity.com/paper/2406.07843/) (2024)
- [Low-Resource Neural Machine Translation Using Recurrent Neural Networks and Transfer Learning: A Case Study on English-to-Igbo](https://mlsanity.com/paper/2504.17252/) (2025)

## Abstract (arXiv)

Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters.