Transformer Approximations from ReLUs
Not on the map. Why?
Abstract (arXiv)
We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 27 references that are in the map. First posted on arXiv 27 Apr 2026.
Builds on
8 of 27 references.
References · 8
- 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
- 2024Transformers, parallel computation, and logarithmic depthcited by 79
- 2024Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theorycited by 63
- 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47
- 2024Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiencycited by 22
- 2025A Theoretical Analysis of Discrete Flow Matching Generative Modelscited by 14
- 2025Universal Approximation with Softmax Attentioncited by 6
- 2025The Transformer Cookbookcited by 6
Built on by
Not cited yet.