mlsanityData to 7 Oct 2026

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

arXiv 2604.01591 ↗First posted 2 Apr 2026cs.AIcited by 5

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 51 of its 53 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

51 of 53 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection29 Sep
  2. 2025Boosting LLM Reasoning via Spontaneous Self-Correction7 Jun
  3. 2025Incentivizing LLMs to Self-Verify Their Answers2 Jun
  4. 2025Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking25 Mar
  5. 2025Self-rewarding correction for mathematical reasoning26 Feb
  6. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision25 Nov
  7. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  8. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning26 Apr
  9. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection15 Mar
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
  12. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 39

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
27 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2025Group Sequence Policy Optimizationcited by 788
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  6. 2025Process Reinforcement through Implicit Rewardscited by 478
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  10. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  11. 2022Generating Sequences by Learning to Self-Correctcited by 271
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  13. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  14. 2024Free Process Rewards without Process Labelscited by 162
  15. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  16. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  17. 2025Geometric-Mean Policy Optimizationcited by 107
  18. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
  19. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  20. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  21. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  22. 2025Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewardscited by 15
  23. 2023Think Twice: A Human-like Two-stage Conversational Agent for Emotional Response Generationcited by 14
  24. 2025Training Language Model to Critique for Better Refinementcited by 7
  25. 2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
  26. 2023Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Modelscited by 6
  27. 2025RISE: Reasoning Enhancement via Iterative Self-Exploration in Multi-hop Question Answeringcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026SafeGEO: Understanding Generative Engine Optimization Risks in Recommendation Agentscited by 4
  2. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branchescited by 1
  3. 2026MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulationscited by 1
  4. 2026Tandem Reinforcement Learning with Verifiable Rewardscited by 0
  5. 2026RAISE: Reinforcing Access Control Policy Synthesis in LLMs via Symbolic Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
  2. 2025Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization
  3. 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models
  4. 2026Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards
  5. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  6. 2025Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLM
  7. 2025Optimizing Length Compression in Large Reasoning Models
  8. 2025Thinkless: LLM Learns When to Think