mlsanityData to 14 Oct 2026

RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory

arXiv 2609.28625 ↗First posted 23 Sep 2026cs.LG cond-mat.dis-nn stat.MLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Despite the importance of reinforcement learning with verifiable rewards (RLVR), the extent to which it can learn new reasoning capabilities remains debated. Here we study the optimization landscape of RLVR on algorithmic tasks, such as iterated group and quasigroup multiplication. To this end, we map entropy-regularized RLVR over myopic tabular policies onto an energy-based (spin-glass) model over deterministic policies. This mapping upper-bounds what RLVR can achieve, and lets us rigorously characterize the landscape in this tabular setting. We show, both theoretically and experimentally, that for a wide class of models and tasks with uncorrelated inputs, this landscape is benign, containing no local minima that could trap RLVR training. Rather, the practical difficulty of these tasks appears to stem, at least in part, from issues such as diffusive barriers and gradient-estimation error in traversing the landscape. These are genuine obstacles that can prevent a solution from being found, but they are distinct from the landscape itself being rugged. We show that these obstacles can often be mitigated through the choice of entropy regulator. Consistent with this theory, we find that a transformer trained from scratch, using only last-token rewards, successfully learns an algorithmic chain of thought for iterated non-Abelian group multiplications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 55 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

32 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR7 Jun
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models8 Nov
  3. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense8 Oct
  4. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models24 Sep
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 23

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  5. 2014The Loss Surfaces of Multilayer Networkscited by 280
  6. 2022Transformers Learn Shortcuts to Automatacited by 268
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  8. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  9. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  11. 2024The Illusion of State in State-Space Modelscited by 187
  12. 2018Understanding the impact of entropy on policy optimizationcited by 176
11 more
  1. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  2. 2023The Quantization Model of Neural Scalingcited by 138
  3. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  4. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  5. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  6. 2025How Reinforcement Learning After Next-Token Prediction Facilitates Learningcited by 10
  7. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 10
  8. 2026The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RLcited by 6
  9. 2026Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimatorcited by 4
  10. 2018Glassy Phase of Optimal Quantum Controlcited by 3
  11. 2026Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradientscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
  2. 2025The Path Not Taken: RLVR Provably Learns Off the Principals
  3. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  4. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  5. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards
  6. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
  7. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
  8. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors