# Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

arXiv: https://arxiv.org/abs/2510.26109 · HTML: https://mlsanity.com/paper/2510.26109/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models](https://mlsanity.com/paper/2508.10751/) (2025)
- [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025)
- [Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions](https://mlsanity.com/paper/2506.07527/) (2025)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [Outcome-based Exploration for LLM Reasoning](https://mlsanity.com/paper/2509.06941/) (2025)
- [Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination](https://mlsanity.com/paper/2507.10532/) (2025)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [Towards a Unified View of Large Language Model Post-Training](https://mlsanity.com/paper/2509.04419/) (2025)
- [Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse](https://mlsanity.com/paper/2509.25808/) (2025)
- [Nudging the Boundaries of LLM Reasoning](https://mlsanity.com/paper/2509.25666/) (2025)
- [BroRL: Scaling Reinforcement Learning via Broadened Exploration](https://mlsanity.com/paper/2510.01180/) (2025)
- [Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation](https://mlsanity.com/paper/2509.25849/) (2025)
- [Improving RL Exploration for LLM Reasoning through Retrospective Replay](https://mlsanity.com/paper/2504.14363/) (2025)
- [A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning](https://mlsanity.com/paper/2507.14295/) (2025)
- [LaSeR: Reinforcement Learning with Last-Token Self-Rewarding](https://mlsanity.com/paper/2510.14943/) (2025)
- [EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning](https://mlsanity.com/paper/2508.07809/) (2025)
- [Learning from Failures in Multi-Attempt Reinforcement Learning](https://mlsanity.com/paper/2503.04808/) (2025)
- [Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought](https://mlsanity.com/paper/2509.04027/) (2025)

## Built on by (most cited first, 3 of 3)

- [No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning](https://mlsanity.com/paper/2601.06794/) (2026)
- [Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning](https://mlsanity.com/paper/2602.22751/) (2026)
- [Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR](https://mlsanity.com/paper/2609.37825/) (2026)

## Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has significantly boosted the reasoning capability of language models (LMs). However, existing RLVR approaches train LMs based on their own on-policy responses and are constrained by the initial capability of LMs, thus prone to exploration stagnation, in which LMs fail to solve more training problems and cannot further learn from the training data. Some approaches try to address this by leveraging off-policy solutions to training problems, but rely on external expert guidance that is limited in availability and scalability. In this work, we propose LTE (Learning to reason from Trial and Error), an approach that hints LMs with their previously self-made mistakes, not requiring any external expert guidance. Experiments validate the effectiveness of LTE, which outperforms the normal group relative policy optimization (GRPO) by 5.02 in Pass@1 and 9.96 in Pass@k on average across six mathematical reasoning benchmarks for Qwen3-8B-Base and even performs better than methods that require external guidance. Further analysis confirms that LTE successfully mitigates exploration stagnation and enhances both exploitation and exploration during training. Our code is available at https://github.com/JamyDon/LTE.