mlsanityData to 14 Oct 2026

Unlocking Recursive Thinking of LLMs: Alignment via Refinement

arXiv 2506.06009 ↗First posted 6 Jun 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

The OpenAI o1-series models have demonstrated that leveraging long-form Chain of Thought (CoT) can substantially enhance performance. However, the recursive thinking capabilities of Large Language Models (LLMs) remain limited, particularly in the absence of expert-curated data for distillation. In this paper, we propose AvR: Alignment via Refinement, a novel method aimed at unlocking the potential of LLMs for recursive reasoning through long-form CoT. AvR introduces a refinement process that integrates criticism and improvement actions, guided by differentiable learning techniques to optimize refinement-aware rewards. As a result, the synthesized multi-round data can be organized as a long refinement thought, further enabling test-time scaling. Experimental results show that AvR significantly outperforms conventional preference optimization methods. Notably, with only 3k synthetic samples, our method boosts the performance of the LLaMA-3-8B-Instruct model by over 20% in win rate on AlpacaEval 2.0. Our code is available at Github (https://github.com/Banner-Z/AvR.git).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 28 of its 48 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

28 of 48 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025s1: Simple test-time scaling31 Jan
  2. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  4. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correction19 Dec
  5. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems12 Dec
  6. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions21 Nov
  7. 2024O1 Replication Journey: A Strategic Progress Report -- Part 18 Oct
  8. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  5. 2024Self-Rewarding Language Modelscited by 681
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  10. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  12. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
7 more
  1. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  2. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  3. 2024Critique-out-Loud Reward Modelscited by 106
  4. 2024o1-Coder: an o1 Replication for Codingcited by 78
  5. 2024DRT: Deep Reasoning Translation via Long Chain-of-Thoughtcited by 29
  6. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 26
  7. 2024Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridgingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Teaching LLMs to Refine with Tools
  2. 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignment
  3. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
  4. 2025From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs
  5. 2025Rectifying LLM Thought from Lens of Optimization
  6. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
  7. 2023Making Large Language Models Better Reasoners with Alignment
  8. 2026Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework