# Competition Dynamics Shape Algorithmic Phases of In-Context Learning

arXiv: https://arxiv.org/abs/2412.01003 · HTML: https://mlsanity.com/paper/2412.01003/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2024.

## Built on by (most cited first, 30 of 54)

- [ICLR: In-Context Learning of Representations](https://mlsanity.com/paper/2501.00070/) (2024)
- [Training Dynamics of In-Context Learning in Linear Attention](https://mlsanity.com/paper/2501.16265/) (2025)
- [How do language models learn facts? Dynamics, curricula and hallucinations](https://mlsanity.com/paper/2503.21676/) (2025)
- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts](https://mlsanity.com/paper/2505.17127/) (2025)
- [Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering](https://mlsanity.com/paper/2511.00617/) (2025)
- [The emergence of sparse attention: impact of data distribution and benefits of repetition](https://mlsanity.com/paper/2505.17863/) (2025)
- [Strategy Coopetition Explains the Emergence and Transience of In-Context Learning](https://mlsanity.com/paper/2503.05631/) (2025)
- [Dynamics of Transient Structure in In-Context Linear Regression Transformers](https://mlsanity.com/paper/2501.17745/) (2025)
- [Do different prompting methods yield a common task representation in language models?](https://mlsanity.com/paper/2505.12075/) (2025)
- [Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalization](https://mlsanity.com/paper/2412.04619/) (2024)
- [Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence](https://mlsanity.com/paper/2505.16694/) (2025)
- [Decomposing Elements of Problem Solving: What "Math" Does RL Teach?](https://mlsanity.com/paper/2505.22756/) (2025)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [Context Structure Reshapes the Representational Geometry of Language Models](https://mlsanity.com/paper/2601.22364/) (2026)
- [Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls](https://mlsanity.com/paper/2510.00184/) (2025)
- [Predicting the Emergence of Induction Heads in Language Model Pretraining](https://mlsanity.com/paper/2511.16893/) (2025)
- [When Is Collective Intelligence a Lottery? Multi-Agent Scaling Laws for Memetic Drift in LLMs](https://mlsanity.com/paper/2603.24676/) (2026)
- [Distinct Computations Emerge From Compositional Curricula in In-Context Learning](https://mlsanity.com/paper/2506.13253/) (2025)
- [In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly](https://mlsanity.com/paper/2506.19351/) (2025)
- [In-Context Algebra](https://mlsanity.com/paper/2512.16902/) (2025)
- [Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descent](https://mlsanity.com/paper/2510.10425/) (2025)
- [Distinct mechanisms underlying in-context learning in transformers](https://mlsanity.com/paper/2604.12151/) (2026)
- [High-Dimensional Statistics: Reflections on Progress and Open Problems](https://mlsanity.com/paper/2605.05076/) (2026)
- [Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning](https://mlsanity.com/paper/2505.11004/) (2025)
- [Latent Concept Disentanglement in Transformer-based Language Models](https://mlsanity.com/paper/2506.16975/) (2025)
- [Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics](https://mlsanity.com/paper/2508.11017/) (2025)
- [Asymptotic Study of In-context Learning with Random Transformers through Equivalent Models](https://mlsanity.com/paper/2509.15152/) (2025)
- [Context and Diversity Matter: The Emergence of In-Context Learning in World Models](https://mlsanity.com/paper/2509.22353/) (2025)
- [Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization](https://mlsanity.com/paper/2510.16096/) (2025)

## Abstract (arXiv)

In-Context Learning (ICL) has significantly expanded the general-purpose nature of large language models, allowing them to adapt to novel tasks using merely the inputted context. This has motivated a series of papers that analyze tractable synthetic domains and postulate precise mechanisms that may underlie ICL. However, the use of relatively distinct setups that often lack a sequence modeling nature to them makes it unclear how general the reported insights from such studies are. Motivated by this, we propose a synthetic sequence modeling task that involves learning to simulate a finite mixture of Markov chains. As we show, models trained on this task reproduce most well-known results on ICL, hence offering a unified setting for studying the concept. Building on this setup, we demonstrate we can explain a model's behavior by decomposing it into four broad algorithms that combine a fuzzy retrieval vs. inference approach with either unigram or bigram statistics of the context. These algorithms engage in a competition dynamics to dominate model behavior, with the precise experimental conditions dictating which algorithm ends up superseding others: e.g., we find merely varying context size or amount of training yields (at times sharp) transitions between which algorithm dictates the model behavior, revealing a mechanism that explains the transient nature of ICL. In this sense, we argue ICL is best thought of as a mixture of different algorithms, each with its own peculiarities, instead of a monolithic capability. This also implies that making general claims about ICL that hold universally across all settings may be infeasible.