mlsanityData to 7 Oct 2026

Solving Empirical Bayes via Transformers

arXiv 2502.09844 ↗First posted 14 Feb 2025cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

This work applies modern AI tools (transformers) to solving one of the oldest statistical problems: Poisson means under empirical Bayes (Poisson-EB) setting. In Poisson-EB a high-dimensional mean vector θ (with iid coordinates sampled from an unknown prior π) is estimated on the basis of X=Poisson(θ). A transformer model is pre-trained on a set of synthetically generated pairs (X,θ) and learns to do in-context learning (ICL) by adapting to unknown π. Theoretically, we show that a sufficiently wide transformer can achieve vanishing regret with respect to an oracle estimator who knows π as dimension grows to infinity. Practically, we discover that already very small models (100k parameters) are able to outperform the best classical algorithm (non-parametric maximum likelihood, or NPMLE) both in runtime and validation loss, which we compute on out-of-distribution synthetic data as well as real-world datasets (NHL hockey, MLB baseball, BookCorpusOpen). Finally, by using linear probes, we confirm that the transformer's EB estimator appears to internally work differently from either NPMLE or Robbins' estimators.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

50 of 80 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
38 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2019The frontier of simulation-based inferencecited by 490
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  8. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  9. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  10. 2022Exploring Length Generalization in Large Language Modelscited by 204
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  12. 2019Universal Approximation with Deep Narrow Networkscited by 198
  13. 2023The emergence of clusters in self-attention dynamicscited by 133
  14. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  15. 2020Minimum Width for Universal Approximationcited by 92
  16. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  17. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  18. 2023In-Context Learning through the Bayesian Prismcited by 78
  19. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  20. 2024All-in-one simulation-based inferencecited by 71
  21. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  22. 2023Learning Robust Statistics for Simulation-based Inference under Model Misspecificationcited by 48
  23. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  24. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  25. 2024Transformers are Universal In-context Learnerscited by 42
  26. 2024Neural Methods for Amortized Inferencecited by 36
  27. 2024Amortized Probabilistic Conditioning for Optimization, Simulation and Inferencecited by 22
  28. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16
  29. 2024A Survey on Universal Approximation Theoremscited by 15
  30. 2021Sharp regret bounds for empirical Bayes and compound decision problemscited by 8
  31. 2022A flexible empirical Bayes approach to multiple linear regression and connections with penalized regressioncited by 8
  32. 2024Transformers are Expressive, But Are They Expressive Enough for Regression?cited by 6
  33. 2024Bayes' Power for Explaining In-Context Learning Generalizationscited by 6
  34. 2020Invidious Comparisons: Ranking and Selection as Compound Decisionscited by 5
  35. 2022Optimal empirical Bayes estimation for the Poisson model via minimum-distance methodscited by 5
  36. 2023A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regressioncited by 4
  37. 2024Generative Bayesian Computation for Maximum Expected Utilitycited by 3
  38. 2024Empirical Bayes for the Reluctant Frequentistcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025TabPFN: One Model to Rule Them All?cited by 10
  2. 2025Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixturescited by 2
  3. 2026InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimatecited by 1
  4. 2026Inverse Problems Conditioned on Observation Ensembles: Applications and Methodscited by 0
  5. 2026LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimizationcited by 0
  6. 2026Universal priors: solving empirical Bayes via Bayesian inference and pretrainingcited by 0
  7. 2026DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passingcited by 0