mlsanityData to 14 Oct 2026

Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-Context

arXiv 2603.10573 ↗First posted 11 Mar 2026cs.LG cs.AIcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

In-context learning (ICL) allows Transformers to adapt to novel tasks without weight updates, yet the underlying algorithms remain poorly understood. We adopt a statistical decision-theoretic perspective by investigating simple binary hypothesis testing, where the optimal policy is determined by the likelihood-ratio test. Notably, this setup provides a mathematically rigorous setting for mechanistic interpretability where the target algorithmic ground truth is known. By training Transformers on tasks requiring distinct geometries (linear shifted means vs. nonlinear variance estimation), we demonstrate that the models approximate the Bayes-optimal sufficient statistics from context up to some monotonic transformation, matching the performance of an ideal oracle estimator in nonlinear regimes. Leveraging this analytical ground truth, mechanistic analysis via logit lens and circuit alignment suggests that the model does not rely on a fixed kernel smoothing heuristic. Instead, it appears to adapt the point at which decisions become linearly decodable: exhibiting patterns consistent with a voting-style ensemble for linear tasks while utilizing a deeper sequential computation for nonlinear tasks. These findings suggest that ICL emerges from the construction of task-adaptive statistical estimators rather than simple similarity matching.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 12 of its 17 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

12 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Progress measures for grokking via mechanistic interpretability12 Jan

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  8. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  9. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  10. 2016Towards a Neural Statisticiancited by 121
  11. 2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization
  2. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
  3. 2024Context-Scaling versus Task-Scaling in In-Context Learning
  4. 2023The Transient Nature of Emergent In-Context Learning in Transformers
  5. 2026Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
  6. 2025Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
  7. 2026Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
  8. 2026Bayesian Optimality of In-Context Learning with Selective State Spaces