# Transformers are Provably Optimal In-context Estimators for Wireless Communications

arXiv: https://arxiv.org/abs/2311.00226 · HTML: https://mlsanity.com/paper/2311.00226/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Transformers learn to implement preconditioned gradient descent for in-context learning](https://mlsanity.com/paper/2306.00297/) (2023)
- [One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention](https://mlsanity.com/paper/2307.03576/) (2023)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [In-Context Convergence of Transformers](https://mlsanity.com/paper/2310.05249/) (2023)
- [How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?](https://mlsanity.com/paper/2310.08391/) (2023)
- [In-Context Learning through the Bayesian Prism](https://mlsanity.com/paper/2306.04891/) (2023)
- [How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations](https://mlsanity.com/paper/2310.10616/) (2023)
- [End-to-end Learning for OFDM: From Neural Receivers to Pilotless Communication](https://mlsanity.com/paper/2009.05261/) (2020)
- [Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms](https://mlsanity.com/paper/2007.13290/) (2020)
- [Unsupervised Linear and Nonlinear Channel Equalization and Decoding using Variational Autoencoders](https://mlsanity.com/paper/1905.08795/) (2019)
- [Variational Autoencoder Leveraged MMSE Channel Estimation](https://mlsanity.com/paper/2205.05345/) (2022)
- [Semi-Supervised Variational Inference over Nonlinear Channels](https://mlsanity.com/paper/2309.11841/) (2023)

## Built on by (most cited first, 8 of 8)

- [DenoMAE2.0: Improving Denoising Masked Autoencoders by Classifying Local Patches](https://mlsanity.com/paper/2502.18202/) (2025)
- [Analogical Learning for Cross-Scenario Generalization: Framework and Application to Intelligent Localization](https://mlsanity.com/paper/2504.08811/) (2025)
- [Decision Feedback In-Context Symbol Detection over Block-Fading Channels](https://mlsanity.com/paper/2411.07600/) (2024)
- [In-Context Learning for Gradient-Free Receiver Adaptation: Principles, Applications, and Theory](https://mlsanity.com/paper/2506.15176/) (2025)
- [Decision Feedback In-Context Learning for Wireless Symbol Detection](https://mlsanity.com/paper/2503.16594/) (2025)
- [Chain-of-Thought Enhanced Shallow Transformers for Wireless Symbol Detection](https://mlsanity.com/paper/2506.21093/) (2025)
- [Self-Evolving In-Context Learning for Direct Pilot-to-Beamformer Design in MU-MISO Systems](https://mlsanity.com/paper/2607.11970/) (2026)
- [Geometric Inductive Biases for Semi-Supervised Equalization: The Constellation-Aware Transformer](https://mlsanity.com/paper/2609.33695/) (2026)

## Abstract (arXiv)

Pre-trained transformers exhibit the capability of adapting to new tasks through in-context learning (ICL), where they efficiently utilize a limited set of prompts without explicit model optimization. The canonical communication problem of estimating transmitted symbols from received observations can be modeled as an in-context learning problem: received observations are a noisy function of transmitted symbols, and this function can be represented by an unknown parameter whose statistics depend on an unknown latent context. This problem, which we term in-context estimation (ICE), has significantly greater complexity than the extensively studied linear regression problem. The optimal solution to the ICE problem is a non-linear function of the underlying context. In this paper, we prove that, for a subclass of such problems, a single-layer softmax attention transformer (SAT) computes the optimal solution of the above estimation problem in the limit of large prompt length. We also prove that the optimal configuration of such a transformer is indeed the minimizer of the corresponding training loss. Further, we empirically demonstrate the proficiency of multi-layer transformers in efficiently solving broader in-context estimation problems. Through extensive simulations, we show that solving ICE problems using transformers significantly outperforms standard approaches. Moreover, just with a few context examples, it achieves the same performance as an estimator with perfect knowledge of the latent context. The code is available hrefhttps://github.com/vishnutez/in-context-estimationhere.