# Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

arXiv: https://arxiv.org/abs/2610.07074 · HTML: https://mlsanity.com/paper/2610.07074/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 45 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 11 of 11)

- [Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers](https://mlsanity.com/paper/2409.10559/) (2024)
- [The Statistical Accuracy of Neural Posterior and Likelihood Estimation](https://mlsanity.com/paper/2411.12068/) (2024)
- [TabPFN: One Model to Rule Them All?](https://mlsanity.com/paper/2505.20003/) (2025)
- [On the Emergence of Induction Heads for In-Context Learning](https://mlsanity.com/paper/2511.01033/) (2025)
- [Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning](https://mlsanity.com/paper/2603.01514/) (2026)
- [Meta-Statistical Learning: Supervised Learning of Statistical Estimators](https://mlsanity.com/paper/2502.12088/) (2025)
- [Theoretical guarantees for neural estimators in parametric statistics](https://mlsanity.com/paper/2506.18508/) (2025)
- [Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions](https://mlsanity.com/paper/2603.06248/) (2026)
- [Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks](https://mlsanity.com/paper/2607.11875/) (2026)
- [Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention](https://mlsanity.com/paper/2605.21292/) (2026)
- [Transformers Can Learn Posterior Predictive Distributions In-Context](https://mlsanity.com/paper/2605.26713/) (2026)

## Abstract (arXiv)

Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistical rule in a two-parameter softmax-attention model with a fixed feature and inequality direction. Pretraining uses labeled contexts and their true thresholds; a fresh threshold must be inferred from context alone. Under a large-resolution initialization, constant-step gradient descent on m tasks with n examples each produces a frozen estimator with error widetilde O((mwedge n)-1+N-1) for each fixed interior threshold and every fresh-context size N. The two terms separate finite-pretraining accuracy from fresh-context localization. The mechanism is coordinated parameter divergence: population training calibrates the relative label and feature scores, then increases the attention scale as t1/4, giving population threshold error O(t-1/4). To transfer this mechanism to a fixed finite corpus, we control gradient errors relative to the shrinking directions of progress at successive parameter scales. This certifies a growing training interval without requiring long-time tracking of the population trajectory. We also identify the boundary limitation of the one-head model and explain statistically what a reflected symmetrization could achieve.