mlsanityData to 14 Oct 2026

Prediction Dynamics in Depth-Recurrent Language Models

arXiv 2609.21383 ↗First posted 18 Sep 2026cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Depth-recurrent language models refine predictions through repeated latent updates. Why can intermediate answers agree with the endpoint while their scores continue to change? We derive a sharp margin characterization that decomposes the conservatism of a magnitude bound into common translation, direction relative to the winner, and the pairing of each competitor's update with its score gap. Across Huginn-3.5B and Ouro-1.4B, accounting for update direction and competitor pairing reduces the mean earliest qualifying depth by a further 22.5-34.4% of the total depth beyond translation removal under full answer-text scoring. This retrospective comparison uses completed trajectories. Substantial contributions also occur under label scoring. For shared predictive distributions, we separate common and contrast motion orthogonally and express the common component through candidate-set mass and within-set concentration. Common and contrast energies can attenuate at different rates, allowing a growing preference-change share to coexist with shrinking absolute updates. These findings explain finite-depth answer preservation through the geometry and composition of observed score changes.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

32 of 38 references.

References · 32

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  4. 2018Universal Transformerscited by 647
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  6. 2019Deep Equilibrium Modelscited by 615
  7. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  8. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  10. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  11. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  12. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
20 more
  1. 2022Confident Adaptive Language Modelingcited by 310
  2. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  3. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  4. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  5. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  6. 2023Looped Transformers as Programmable Computerscited by 237
  7. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  8. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  9. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 185
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  11. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  12. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 142
  13. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  14. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  15. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  16. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  17. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  18. 2025Answer Convergence as a Signal for Early Stopping in Reasoningcited by 40
  19. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 30
  20. 2026Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Monitoring latent reasoning in looped language modelscited by 0