# Transformer Approximations from ReLUs

arXiv: https://arxiv.org/abs/2604.24878 · HTML: https://mlsanity.com/paper/2604.24878/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 8 of 8)

- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Transformers, parallel computation, and logarithmic depth](https://mlsanity.com/paper/2402.09268/) (2024)
- [Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theory](https://mlsanity.com/paper/2403.11968/) (2024)
- [Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?](https://mlsanity.com/paper/2307.14023/) (2023)
- [Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency](https://mlsanity.com/paper/2411.16525/) (2024)
- [A Theoretical Analysis of Discrete Flow Matching Generative Models](https://mlsanity.com/paper/2509.22623/) (2025)
- [Universal Approximation with Softmax Attention](https://mlsanity.com/paper/2504.15956/) (2025)
- [The Transformer Cookbook](https://mlsanity.com/paper/2510.00368/) (2025)

## Abstract (arXiv)

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.