# Convolution, attention and structure embedding

arXiv: https://arxiv.org/abs/1905.01289 · HTML: https://mlsanity.com/paper/1905.01289/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2019.

## Built on by (most cited first, 15 of 15)

- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [On the Connection between Local Attention and Dynamic Depth-wise Convolution](https://mlsanity.com/paper/2106.04263/) (2021)
- [GOAL: A Generalist Combinatorial Optimization Agent Learner](https://mlsanity.com/paper/2406.15079/) (2024)
- [Polarized Self-Attention: Towards High-quality Pixel-wise Regression](https://mlsanity.com/paper/2107.00782/) (2021)
- [Structured Initialization for Vision Transformers](https://mlsanity.com/paper/2505.19985/) (2025)
- [DeepGene Transformer: Transformer for the gene expression-based classification of cancer subtypes](https://mlsanity.com/paper/2108.11833/) (2021)
- [Infusing Lattice Symmetry Priors in Attention Mechanisms for Sample-Efficient Abstract Geometric Reasoning](https://mlsanity.com/paper/2306.03175/) (2023)
- [Convolutional Initialization for Data-Efficient Vision Transformers](https://mlsanity.com/paper/2401.12511/) (2024)
- [Structured Initialization for Attention in Vision Transformers](https://mlsanity.com/paper/2404.01139/) (2024)
- [Interpret Vision Transformers as ConvNets with Dynamic Convolutions](https://mlsanity.com/paper/2309.10713/) (2023)
- [Alias-Free ViT: Fractional Shift Invariance via Linear Attention](https://mlsanity.com/paper/2510.22673/) (2025)
- [Structured Time Series Prediction without Structural Prior](https://mlsanity.com/paper/2202.03539/) (2022)
- [On Inductive Biases for Machine Learning in Data Constrained Settings](https://mlsanity.com/paper/2302.10692/) (2023)
- [DMON: A Simple yet Effective Approach for Argument Structure Learning](https://mlsanity.com/paper/2405.01216/) (2024)
- [Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric](https://mlsanity.com/paper/2410.17628/) (2024)

## Abstract (arXiv)

Deep neural networks are composed of layers of parametrised linear operations intertwined with non linear activations. In basic models, such as the multi-layer perceptron, a linear layer operates on a simple input vector embedding of the instance being processed, and produces an output vector embedding by straight multiplication by a matrix parameter. In more complex models, the input and output are structured and their embeddings are higher order tensors. The parameter of each linear operation must then be controlled so as not to explode with the complexity of the structures involved. This is essentially the role of convolution models, which exist in many flavours dependent on the type of structure they deal with (grids, networks, time series etc.). We present here a unified framework which aims at capturing the essence of these diverse models, allowing a systematic analysis of their properties and their mutual enrichment. We also show that attention models naturally fit in the same framework: attention is convolution in which the structure itself is adaptive, and learnt, instead of being given a priori.