mlsanityData to 14 Oct 2026

Value-Gradient Hypothesis of RL for LLMs

arXiv 2605.21654 ↗First posted 20 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains. We develop a value-gradient perspective of critic-free RL for LLM post-training. First, under a differentiable rollout and additive-noise parameterization, we show that the actor update is value-gradient-like in expectation: the backward pass propagates costates whose conditional expectation equals the value gradient. Second, for discrete transformer policies, we show that autodifferentiation through attention produces empirical costates that approximate this value signal, with an error controlled by the sampling gap and policy entropy. These results motivate a decomposition of RL impact into value gradient signal and reachable reward headroom, yielding a criterion for when RL should be most effective along a pretraining trajectory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 10 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

8 of 10 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic1 Mar
  2. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization9 Nov
  3. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments5 Nov
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 4

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  3. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
  4. 2026Reinforcement Learning via Value Gradient Flowcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning for LLM Post-Training: A Survey
  2. 2024How Can LLM Guide RL? A Value-Based Approach
  3. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  4. 2026Demystifying Reinforcement Learning Post-Training of Language Models
  5. 2025RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs
  6. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2026Linear Dynamics in the RLVR Training of Large Language Models