# The Convergence Gap: Instruction-Tuned Language Models Stabilize Later in the Forward Pass

arXiv: https://arxiv.org/abs/2605.07282 · HTML: https://mlsanity.com/paper/2605.07282/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 4 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 3 of 3)

- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models](https://mlsanity.com/paper/2410.20008/) (2024)
- [Calibration Across Layers: Understanding Calibration Evolution in LLMs](https://mlsanity.com/paper/2511.00280/) (2025)

## Abstract (arXiv)

Final outputs hide when a checkpoint commits to its next-token prediction. We introduce the convergence gap, a model-diffing diagnostic that decodes each layer's next-token distribution and measures its distance to the model's own final distribution. Across six paired pretrained and instruction-tuned checkpoints in native prompting regimes, instruction-tuned checkpoints remain farther from their final predictions later into the stack. The effect persists under endpoint-matched raw and tuned readouts, endpoint-free same-history checks, and fixed-history template replay. Matched-prefix interventions identify late MLP windows as the largest tested leverage point: late IT grafts into PT hosts increase late KL by +0.34 nats, while PT-late swaps into IT hosts reduce it by -0.51 nats; matched random late perturbations give only +0.003 versus +0.327 for the true late graft. A preselected Gemma case study provides behavior-facing plausibility for the same late swap, without serving as a benchmark claim. These results identify a robust predictiondynamics signature of post-training: released instruction-following checkpoints tend to settle later, and late MLP computation is the strongest tested bidirectional handle on that delay under matched histories.