mlsanityData to 14 Oct 2026

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

arXiv 2606.03234 ↗First posted 2 Jun 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 36 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

32 of 36 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Step-wise Rubric Rewards for LLM Reasoning17 May
  2. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration2 Nov
  3. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning26 Oct
  4. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization23 Sep
  5. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  6. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control5 Aug
  7. 2025Geometric-Mean Policy Optimization28 Jul
  8. 2025APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization26 Jun
  9. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  10. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  10. 2025Group Sequence Policy Optimizationcited by 835
  11. 2019Contrastive Representation Distillationcited by 748
  12. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
8 more
  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  3. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  4. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signalscited by 24
  5. 2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
  6. 2026Closing the Modality Reasoning Gap for Speech Large Language Modelscited by 9
  7. 2026Leveraging Error Diversity in Group Rollouts for Reinforcement Learningcited by 3
  8. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  3. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  4. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  5. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
  6. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example
  8. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs