# Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs

arXiv: https://arxiv.org/abs/2509.00084 · HTML: https://mlsanity.com/paper/2509.00084/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 20 of 20)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [Generative Reward Models](https://mlsanity.com/paper/2410.12832/) (2024)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning](https://mlsanity.com/paper/2505.10320/) (2025)
- [ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates](https://mlsanity.com/paper/2502.06772/) (2025)
- [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023)
- [Reward Reasoning Model](https://mlsanity.com/paper/2505.14674/) (2025)
- [LLMs Can Generate a Better Answer by Aggregating Their Own Responses](https://mlsanity.com/paper/2503.04104/) (2025)
- [From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning](https://mlsanity.com/paper/2501.11877/) (2025)
- [CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesis](https://mlsanity.com/paper/2501.01668/) (2025)

## Built on by (most cited first, 12 of 12)

- [Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models](https://mlsanity.com/paper/2509.26626/) (2025)
- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System](https://mlsanity.com/paper/2509.24855/) (2025)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling](https://mlsanity.com/paper/2605.25143/) (2026)
- [Learning Generative Selection for Best-of-N](https://mlsanity.com/paper/2602.02143/) (2026)
- [Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Bias](https://mlsanity.com/paper/2604.02923/) (2026)
- [SCATR: Simple Calibrated Test-Time Ranking](https://mlsanity.com/paper/2604.16535/) (2026)
- [VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus](https://mlsanity.com/paper/2608.10665/) (2026)
- [Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation](https://mlsanity.com/paper/2608.18379/) (2026)
- [The Rise of Verbal Reinforcement Learning](https://mlsanity.com/paper/2609.01597/) (2026)
- [Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes](https://mlsanity.com/paper/2609.19156/) (2026)

## Abstract (arXiv)

Test-time scaling (TTS) has gained widespread attention for enhancing LLM reasoning. Existing approaches such as Best-of-N and majority voting are limited as their performance depends on the quality of candidate responses, making them unable to produce a correct solution when all candidates are incorrect. Parallel self-refinement, generating multiple candidates and synthesizing a refined answer conditioned on them, offers a promising alternative, but the underlying mechanism driving its effectiveness remains obscure. To bridge this gap in understanding, we introduce a new metric, the Refinement Gap, designed to quantify the relative improvement of self-refinement beyond majority voting. We show that the Refinement Gap exhibits a clear scaling trend with model size and is only weakly correlated with the base capability. Based on this discovery, we propose Generative Self-Refinement (GSR), a parallel test-time scaling framework that transfers the refinement policy from larger teacher models with higher refinement gap into smaller students. Crucially, GSR jointly trains a single model to generate strong candidates and refine a better final answer based on these candidates. Experimental results demonstrate that our method achieves state-of-the-art performance across five mathematical benchmarks over other parallel aggregation methods, while the learned refinement skill transfers across multiple model scales and families and exhibits robust generalization to an out-of-distribution domain.