# Is In-Context Universality Enough? MLPs are Also Universal In-Context

arXiv: https://arxiv.org/abs/2502.03327 · HTML: https://mlsanity.com/paper/2502.03327/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2025.

## Built on by (most cited first, 4 of 4)

- [Pretrain-Test Task Alignment Governs Generalization in In-Context Learning](https://mlsanity.com/paper/2509.26551/) (2025)
- [Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data](https://mlsanity.com/paper/2509.00924/) (2025)
- [Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning](https://mlsanity.com/paper/2605.04995/) (2026)
- [A Closed-Form Formula for Consistent Lipschitz Regression on Metric Spaces with Sparse Neural Network Realizations](https://mlsanity.com/paper/2609.03129/) (2026)

## Abstract (arXiv)

The success of transformers is often linked to their ability to perform in-context learning. Recent work shows that transformers are universal in context, capable of approximating any real-valued continuous function of a context (a probability measure over Xsubseteq ℝd) and a query x∈ X. This raises the question: Does in-context universality explain their advantage over classical models? We answer this in the negative by proving that MLPs with trainable activation functions are also universal in-context. This suggests the transformer's success is likely due to other factors like inductive bias or training stability.