# Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers

arXiv: https://arxiv.org/abs/2609.31458 · HTML: https://mlsanity.com/paper/2609.31458/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 38 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 3 of 3)

- [Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of-Experts](https://mlsanity.com/paper/2502.13577/) (2025)
- [Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights](https://mlsanity.com/paper/2505.03205/) (2025)
- [A Deep Generative Approach to Stratified Learning](https://mlsanity.com/paper/2604.10650/) (2026)

## Abstract (arXiv)

Transformers have become a central architecture for in-context learning (ICL), particularly through their state-of-the-art performance in large language models. This success motivates understanding how transformers exploit task-relevant structure in geometrically heterogeneous data. However, existing nonparametric ICL theory has largely focused on Euclidean domains or single-manifold models. To address this gap, we study the prediction problem under unknown local geometry, modeled by sample size-dependent mixtures of manifolds with heterogeneous dimensions, smoothness, and sampling masses. Under local separation and small-perturbation conditions, we establish a minimax lower bound capturing the aggregate difficulty of the components and construct an oracle tangent local-polynomial estimator with a matching upper bound. This estimator is connected to a structure-informed, two-stage softmax transformer with a geometric preconditioner and chartwise reduced local-polynomial solvers. The transformer achieves negligible approximation error relative to the minimax rate with logarithmic depth and polynomial size. Finally, we derive an in-context generalization bound for near empirical risk minimizers over this class. Together, these results identify conditions under which the resulting predictor exploits local geometry and attains the aggregate minimax rate.