# The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason

arXiv: https://arxiv.org/abs/2505.22653 · HTML: https://mlsanity.com/paper/2505.22653/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 2 of its 38 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 2 of 2)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)

## Built on by (most cited first, 12 of 12)

- [Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination](https://mlsanity.com/paper/2507.10532/) (2025)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context](https://mlsanity.com/paper/2507.00417/) (2025)
- [Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models](https://mlsanity.com/paper/2509.26114/) (2025)
- [Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR](https://mlsanity.com/paper/2605.02909/) (2026)
- [Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions](https://mlsanity.com/paper/2508.21188/) (2025)
- [Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning](https://mlsanity.com/paper/2512.16917/) (2025)
- [Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2603.16140/) (2026)
- [Sample-efficient LLM Optimization with Reset Replay](https://mlsanity.com/paper/2508.06412/) (2025)
- [Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models](https://mlsanity.com/paper/2605.07244/) (2026)
- [What do Reward Models Memorize?](https://mlsanity.com/paper/2607.24484/) (2026)
- [Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR](https://mlsanity.com/paper/2610.17824/) (2026)

## Abstract (arXiv)

Recent studies on post-training large language models (LLMs) for reasoning through reinforcement learning (RL) typically focus on tasks that can be accurately verified and rewarded, such as solving math problems. In contrast, our research investigates the impact of reward noise, a more practical consideration for real-world scenarios involving the post-training of LLMs using reward models. We found that LLMs demonstrate strong robustness to substantial reward noise. For example, manually flipping 40% of the reward function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve rapid convergence, improving its performance on math tasks from 5% to 72%, compared to the 75% accuracy achieved by a model trained with noiseless rewards. Surprisingly, by only rewarding the appearance of key reasoning phrases (namely reasoning pattern reward, RPR), such as ``first, I need to''-without verifying the correctness of answers, the model achieved peak downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models trained with strict correctness verification and accurate rewards. Recognizing the importance of the reasoning process over the final results, we combined RPR with noisy reward models. RPR helped calibrate the noisy reward models, mitigating potential false negatives and enhancing the LLM's performance on open-ended tasks. These findings suggest the importance of improving models' foundational abilities during the pre-training phase while providing insights for advancing post-training techniques. Our code and scripts are available at https://github.com/trestad/Noisy-Rewards-in-Learning-to-Reason.