# Learning from Failures in Multi-Attempt Reinforcement Learning

arXiv: https://arxiv.org/abs/2503.04808 · HTML: https://mlsanity.com/paper/2503.04808/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 19 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Multi-turn Reinforcement Learning from Preference Human Feedback](https://mlsanity.com/paper/2405.14655/) (2024)
- [Direct Multi-Turn Preference Optimization for Language Agents](https://mlsanity.com/paper/2406.14868/) (2024)
- [Multi-Turn Code Generation Through Single-Step Rewards](https://mlsanity.com/paper/2502.20380/) (2025)
- [HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2203.07376/) (2022)
- [Context-Aware Language Modeling for Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/2204.10198/) (2022)
- [A Context-aware Natural Language Generator for Dialogue Systems](https://mlsanity.com/paper/1608.07076/) (2016)
- [Thinker: Learning to Plan and Act](https://mlsanity.com/paper/2307.14993/) (2023)

## Built on by (most cited first, 6 of 6)

- [When Two LLMs Debate, Both Think They'll Win](https://mlsanity.com/paper/2505.19184/) (2025)
- [Thinker: Learning to Think Fast and Slow](https://mlsanity.com/paper/2505.21097/) (2025)
- [HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.10724/) (2025)
- [Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error](https://mlsanity.com/paper/2510.26109/) (2025)
- [MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learning](https://mlsanity.com/paper/2601.17006/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)

## Abstract (arXiv)

Recent advancements in reinforcement learning (RL) for large language models (LLMs), exemplified by DeepSeek R1, have shown that even a simple question-answering task can substantially improve an LLM's reasoning capabilities. In this work, we extend this approach by modifying the task into a multi-attempt setting. Instead of generating a single response per question, the model is given multiple attempts, with feedback provided after incorrect responses. The multi-attempt task encourages the model to refine its previous attempts and improve search efficiency. Experimental results show that even a small LLM trained on a multi-attempt task achieves significantly higher accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM trained on a standard single-turn task exhibits only a marginal improvement, increasing from 42.3% to 43.2% when given more attempts during evaluation. The results indicate that, compared to the standard single-turn task, an LLM trained on a multi-attempt task achieves slightly better performance on math benchmarks while also learning to refine its responses more effectively based on user feedback. Full code is available at https://github.com/DualityRL/multi-attempt