# Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

arXiv: https://arxiv.org/abs/2606.03234 · HTML: https://mlsanity.com/paper/2606.03234/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs](https://mlsanity.com/paper/2406.10216/) (2024)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening](https://mlsanity.com/paper/2506.02355/) (2025)
- [Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning](https://mlsanity.com/paper/2508.09726/) (2025)
- [NGRPO: Negative-enhanced Group Relative Policy Optimization](https://mlsanity.com/paper/2509.18851/) (2025)
- [LLM-Empowered State Representation for Reinforcement Learning](https://mlsanity.com/paper/2407.13237/) (2024)
- [LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals](https://mlsanity.com/paper/2604.05655/) (2026)
- [GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control](https://mlsanity.com/paper/2508.03772/) (2025)
- [Step-wise Rubric Rewards for LLM Reasoning](https://mlsanity.com/paper/2605.17291/) (2026)
- [Foundation Models for Semantic Novelty in Reinforcement Learning](https://mlsanity.com/paper/2211.04878/) (2022)
- [APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization](https://mlsanity.com/paper/2506.21655/) (2025)
- [Closing the Modality Reasoning Gap for Speech Large Language Models](https://mlsanity.com/paper/2601.05543/) (2026)
- [FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning](https://mlsanity.com/paper/2510.22543/) (2025)
- [Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration](https://mlsanity.com/paper/2511.00794/) (2025)

## Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.