mlsanityData to 14 Oct 2026

CausalLM is not optimal for in-context learning

arXiv 2308.06912 ↗First posted 14 Aug 2023cs.LG cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Recent empirical evidence indicates that transformer based in-context learning performs better when using a prefix language model (prefixLM), in which in-context samples can all attend to each other, compared to causal language models (causalLM), which use auto-regressive attention that prohibits in-context samples to attend to future samples. While this result is intuitive, it is not understood from a theoretical perspective. In this paper we take a theoretical approach and analyze the convergence behavior of prefixLM and causalLM under a certain parameter construction. Our analysis shows that both LM types converge to their stationary points at a linear rate, but that while prefixLM converges to the optimal solution of linear regression, causalLM convergence dynamics follows that of an online gradient descent algorithm, which is not guaranteed to be optimal even as the number of samples grows infinitely. We supplement our theoretical claims with empirical experiments over synthetic and real tasks and using various types of transformers. Our experiments verify that causalLM consistently underperforms prefixLM in all settings.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

23 of 27 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
11 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2022Transformers learn in-context by gradient descentcited by 732
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  5. 2018Universal Transformerscited by 647
  6. 2023Larger language models do in-context learning differentlycited by 394
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  8. 2022UL2: Unifying Language Learning Paradigmscited by 295
  9. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  10. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  11. 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 8; 2026: 4.

26 papers.

Sort:

By area:

  1. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  2. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  3. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  4. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  5. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  6. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  7. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15
  8. 2025TabFlex: Scaling Tabular Learning to Millions with Linear Attentioncited by 15
  9. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 14
  10. 2024Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysiscited by 14
  11. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  12. 2024BERTs are Generative In-Context Learnerscited by 12
14 more
  1. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 12
  2. 2024In-context learning and Occam's razorcited by 8
  3. 2025Rethinking Invariance in In-context Learningcited by 8
  4. 2024Orchid: Flexible and Data-Dependent Convolution for Sequence Modelingcited by 6
  5. 2024ENTP: Encoder-only Next Token Predictioncited by 5
  6. 2026Auto-Regressive Masked Diffusion Modelscited by 4
  7. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  8. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  9. 2026Learning to Adapt: In-Context Learning Beyond Stationaritycited by 2
  10. 2024Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictionscited by 1
  11. 2024Intermittent Semi-Working Mask: A New Masking Paradigm for LLMscited by 0
  12. 2025Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspectivecited by 0
  13. 2026Recurrent Reasoning on Symbolic Puzzles with Sequence Modelscited by 0
  14. 2026Adapting Linear-Time Architectures for Tabular In-Context Learningcited by 0