# A Simple Explanation for the Phase Transition in Large Language Models with List Decoding

arXiv: https://arxiv.org/abs/2303.13112 · HTML: https://mlsanity.com/paper/2303.13112/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Inverse scaling can become U-shaped](https://mlsanity.com/paper/2211.02011/) (2022)
- [A Time-dependent SIR model for COVID-19 with Undetectable Infected Persons](https://mlsanity.com/paper/2003.00122/) (2020)

## Built on by (most cited first, 3 of 3)

- [Reducing Selection Bias in Large Language Models](https://mlsanity.com/paper/2402.01740/) (2024)
- [Generative Criticality in Large Language Model Temperature Scaling](https://mlsanity.com/paper/2606.06238/) (2026)
- [A Mathematical Theory for Learning Semantic Languages by Abstract Learners](https://mlsanity.com/paper/2404.07009/) (2024)

## Abstract (arXiv)

Various recent experimental results show that large language models (LLM) exhibit emergent abilities that are not present in small models. System performance is greatly improved after passing a certain critical threshold of scale. In this letter, we provide a simple explanation for such a phase transition phenomenon. For this, we model an LLM as a sequence-to-sequence random function. Instead of using instant generation at each step, we use a list decoder that keeps a list of candidate sequences at each step and defers the generation of the output sequence at the end. We show that there is a critical threshold such that the expected number of erroneous candidate sequences remains bounded when an LLM is below the threshold, and it grows exponentially when an LLM is above the threshold. Such a threshold is related to the basic reproduction number in a contagious disease.