# Universal Approximation with Softmax Attention

arXiv: https://arxiv.org/abs/2504.15956 · HTML: https://mlsanity.com/paper/2504.15956/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2025.

## Built on by (most cited first, 6 of 6)

- [A unified framework for establishing the universal approximation of transformer-type architectures](https://mlsanity.com/paper/2506.23551/) (2025)
- [Accelerating Benchmarking of Functional Connectivity Modeling via Structure-aware Core-set Selection](https://mlsanity.com/paper/2602.05667/) (2026)
- [Dynamic Relational Priming Improves Transformer in Multivariate Time Series](https://mlsanity.com/paper/2509.12196/) (2025)
- [On the Universality of Transformer Architectures; How Much Attention Is Enough?](https://mlsanity.com/paper/2512.18445/) (2025)
- [Transformer Approximations from ReLUs](https://mlsanity.com/paper/2604.24878/) (2026)
- [Universal interpolation for deep residual self-attention networks](https://mlsanity.com/paper/2610.01981/) (2026)

## Abstract (arXiv)

We prove that with linear transformations, both (i) two-layer self-attention and (ii) one-layer self-attention followed by a softmax function are universal approximators for continuous sequence-to-sequence functions on compact domains. Our main technique is a new interpolation-based method for analyzing attention's internal mechanism. This leads to our key insight: self-attention is able to approximate a generalized version of ReLU to arbitrary precision, and hence subsumes many known universal approximators. Building on these, we show that two-layer multi-head attention alone suffices as a sequence-to-sequence universal approximator. In contrast, prior works rely on feed-forward networks to establish universal approximation in Transformers. Furthermore, we extend our techniques to show that, (softmax-)attention-only layers are capable of approximating various statistical models in-context. We believe these techniques hold independent interest.