# Breaking through the learning plateaus of in-context learning in Transformer arXiv: https://arxiv.org/abs/2309.06054 ยท HTML: https://mlsanity.com/paper/2309.06054/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 39 of its 52 references that are in the map. - First posted on arXiv 12 Sep 2023. ## Builds on (most cited first, 30 of 39) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023) - [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022) - [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016) - [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022) - [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022) - [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022) - [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018) - [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021) - [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022) - [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023) - [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022) - [Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection](https://mlsanity.com/paper/2306.04637/) (2023) - [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022) - [Looped Transformers as Programmable Computers](https://mlsanity.com/paper/2301.13196/) (2023) - [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020) - [Transformers as Algorithms: Generalization and Stability in In-context Learning](https://mlsanity.com/paper/2301.07067/) (2023) - [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://mlsanity.com/paper/2212.10559/) (2022) - [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022) - [Birth of a Transformer: A Memory Viewpoint](https://mlsanity.com/paper/2306.00802/) (2023) - [Inductive Biases and Variable Creation in Self-Attention Mechanisms](https://mlsanity.com/paper/2110.10090/) (2021) - [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019) ## Built on by (most cited first, 8 of 8) - [When can in-context learning generalize out of task distribution?](https://mlsanity.com/paper/2506.05574/) (2025) - [Task Diversity Shortens the ICL Plateau](https://mlsanity.com/paper/2410.05448/) (2024) - [Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation](https://mlsanity.com/paper/2604.18223/) (2026) - [Layer-Parallel Training for Transformers](https://mlsanity.com/paper/2601.09026/) (2026) - [Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Search](https://mlsanity.com/paper/2605.04057/) (2026) - [The Essence of Balance for Self-Improving Agents in Vision-and-Language Navigation](https://mlsanity.com/paper/2604.19064/) (2026) - [Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data](https://mlsanity.com/paper/2506.14704/) (2025) - [Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatedness](https://mlsanity.com/paper/2604.01601/) (2026) ## Abstract (arXiv) In-context learning, i.e., learning from context examples, is an impressive ability of Transformer. Training Transformers to possess this in-context learning skill is computationally intensive due to the occurrence of learning plateaus, which are periods within the training process where there is minimal or no enhancement in the model's in-context learning capability. To study the mechanism behind the learning plateaus, we conceptually seperate a component within the model's internal representation that is exclusively affected by the model's weights. We call this the "weights component", and the remainder is identified as the "context component". By conducting meticulous and controlled experiments on synthetic tasks, we note that the persistence of learning plateaus correlates with compromised functionality of the weights component. Recognizing the impaired performance of the weights component as a fundamental behavior drives learning plateaus, we have developed three strategies to expedite the learning of Transformers. The effectiveness of these strategies is further confirmed in natural language processing tasks. In conclusion, our research demonstrates the feasibility of cultivating a powerful in-context learning ability within AI systems in an eco-friendly manner.