mlsanityData to 7 Oct 2026

Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning

arXiv 2605.29782 ↗First posted 28 May 2026cs.LG cs.AI cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

36 of 49 references.

References · 36

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
24 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  10. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  11. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  12. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  13. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  14. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  15. 2024Free Process Rewards without Process Labelscited by 162
  16. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  17. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  18. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  19. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  20. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  21. 2019Successor Options: An Option Discovery Framework for Reinforcement Learningcited by 23
  22. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  23. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  24. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026APeB: Benchmarking Personalization Ability of Large Language Model Agentscited by 0
  2. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVRcited by 0