mlsanityData to 7 Oct 2026

Transformers Handle Endogeneity in In-Context Linear Regression

arXiv 2410.01265 ↗First posted 2 Oct 2024stat.ML cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares (textsf2SLS) solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the textsf2SLS method, in the presence of endogeneity.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

32 of 49 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
20 more
  1. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  2. 2023Looped Transformers as Programmable Computerscited by 224
  3. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  4. 2022Multi-Game Decision Transformerscited by 212
  5. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  6. 2019Kernel Instrumental Variable Regressioncited by 118
  7. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  8. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  9. 2023In-Context Convergence of Transformerscited by 112
  10. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  11. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 51
  12. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  13. 2024Transformers on Markov Data: Constant Depth Sufficescited by 26
  14. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 23
  15. 2023Can a Transformer Represent a Kalman Filter?cited by 20
  16. 2024Local to Global: Learning Dynamics and Effect of Initialization for Transformerscited by 11
  17. 2023Stochastic Online Instrumental Variable Regression: Regrets for Endogeneity and Bandit Feedbackcited by 7
  18. 2024Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradientscited by 6
  19. 2024Meta-Learning with Generalized Ridge Regression: High-dimensional Asymptotics, Optimality and Hyper-covariance Estimationcited by 5
  20. 2024Stochastic Optimization Algorithms for Instrumental Variable Regression with Streaming Datacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  2. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvementcited by 1
  3. 2025How Private is Your Attention? Bridging Privacy with In-Context Learningcited by 0
  4. 2025Interpretability and Generalization Bounds for Learning Spatial Physicscited by 0
  5. 2025Differentially Private Two-Stage Gradient Descent for Instrumental Variable Regressioncited by 0
  6. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  7. 2026Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?cited by 0
  8. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0