# Mpemba Effect in Large-Language Model Training Dynamics: A Minimal Analysis of the Valley-River model

arXiv: https://arxiv.org/abs/2507.04206 · HTML: https://mlsanity.com/paper/2507.04206/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 6 Jul 2025.

## Builds on (most cited first, 6 of 6)

- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective](https://mlsanity.com/paper/2410.05192/) (2024)
- [Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions](https://mlsanity.com/paper/2206.01246/) (2022)
- [Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training](https://mlsanity.com/paper/2410.23922/) (2024)
- [Neural Thermodynamic Laws for Large Language Model Training](https://mlsanity.com/paper/2505.10559/) (2025)
- [FOCUS: First Order Concentrated Updating Scheme](https://mlsanity.com/paper/2501.12243/) (2025)

## Abstract (arXiv)

Learning rate (LR) schedules in large language model (LLM) training often follow empirical templates: warm-up, constant plateau/stable phase, and decay (WSD). However, the mechanistic explanation for this strategy remains underexplored, and the choice of plateau height and decay schedule is largely heuristic. In this paper, we connect training dynamics to a thermodynamic analogy via the Mpemba effect - a phenomenon in which a hotter system cools faster than a colder one when quenched into the same bath. We analyze a class of "valley-river" loss landscapes, where sharp (valley) directions equilibrate quickly, while flatter (river) directions govern global descent. The Mpemba effect provides an explanation for the necessity of the warm-up phase and motivates a high plateau - rather than a low one - for accelerating loss decrease during decay. We show that for certain loss landscapes, there exists an optimal plateau learning rate - the "strong Mpemba point" - at which the slowest mode vanishes, resulting in faster convergence during the decay phase. We derive analytical conditions for its existence and estimate decay dynamics required to preserve the Mpemba advantage. Our minimal model and analysis offer a principled justification for plateau-based schedulers and provide guidance for tuning LR in LLMs with minimal hyperparameter sweep.