mlsanityData to 14 Oct 2026

On Predictability of Reinforcement Learning Dynamics for Large Language Models

arXiv 2510.00553 ↗First posted 1 Oct 2025cs.LG cs.AIcited by 23

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in reasoning capabilities of large language models (LLMs) are largely driven by reinforcement learning (RL), yet the underlying parameter dynamics during RL training remain poorly understood. This work identifies two fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1 Dominance, where the top singular subspace of the parameter update matrix nearly fully determines reasoning improvements, recovering over 99% of performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace evolves linearly throughout training, enabling accurate prediction from early checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the generalizability of these properties. More importantly, based on these findings, we propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window, achieving up to 2.5 speedup while retaining textgreater 96% of reasoning performance without extra modules or hyperparameter tuning. This positions our finding as a versatile and practical tool for large-scale RL, opening a path toward principled, interpretable, and efficient training paradigm for LLMs.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 36 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

27 of 36 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling18 Apr
  6. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2023Let's Verify Step by Step31 May
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  10. 2025LIMO: Less is More for Reasoningcited by 498
  11. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  12. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 129
5 more
  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  2. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 100
  3. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
  4. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  5. 2015The Singular Value Decomposition, Applications and Beyondcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 23.

23 papers.

Sort:

By area:

  1. 2026Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillationcited by 19
  2. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  3. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  4. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  5. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 5
  6. 2026Linear Dynamics in the RLVR Training of Large Language Modelscited by 4
  7. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  8. 2026You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectoriescited by 3
  9. 2026On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVRcited by 2
  10. 2026Reasoning Can Be Restored by Correcting a Few Decision Tokenscited by 2
  11. 2026Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RLcited by 2
  12. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMscited by 2
11 more
  1. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  2. 2026Spectral Rewiring for Exploration, Purification, and Model Mergingcited by 1
  3. 2026FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Modelscited by 0
  4. 2026Gradient Extrapolation-Based Policy Optimizationcited by 0
  5. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Modelscited by 0
  6. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  7. 2026Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Mergingcited by 0
  8. 2026Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectorscited by 0
  9. 2026The Low-Rank Structure of VLA Reinforcement Learningcited by 0
  10. 2026Does Scaling Reinforcement Learning Really Require More Training?cited by 0
  11. 2026Learning to Revise Reasoning with Segment-wise On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
  2. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  3. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  4. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  5. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
  6. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  8. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions