mlsanityData to 14 Oct 2026

When Context Sticks: Studying Interference in In-Context Learning

arXiv 2604.23371 ↗First posted 25 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear and quadratic functions, we examine how models trained under sequential, mixed, and random curricula handle abrupt task switches during inference. By sweeping over structured combinations of misleading linear examples followed by recovery quadratic examples, we quantify how prior context biases prediction error and how quickly models realign. Our results show strong evidence of persistent interference: more preceding linear examples reliably degrade quadratic predictions, while additional quadratic examples reduce error but with diminishing returns. We further find that training curricula significantly modulate resilience, with sequential training on the target function class yielding the fastest recovery, and surprisingly, random training producing the least robust behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 25 references that are in the map. First posted on arXiv 25 Apr 2026.

Builds on

22 of 25 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  11. 2023Function Vectors in Large Language Modelscited by 308
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
10 more
  1. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  2. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  3. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  4. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  5. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  6. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  7. 2024Revisiting In-Context Learning with Long Context Language Modelscited by 8
  8. 2023In-context Interference in Chat-based Large Language Modelscited by 3
  9. 2025Order Matters: Rethinking Prompt Construction in In-Context Learningcited by 3
  10. 2024How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classescited by 2

Built on by

Not cited yet.