# ThinkTuning: Instilling Cognitive Reflections without Distillation

arXiv: https://arxiv.org/abs/2508.07616 · HTML: https://mlsanity.com/paper/2508.07616/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Kickstarting Deep Reinforcement Learning](https://mlsanity.com/paper/1803.03835/) (2018)
- [PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving](https://mlsanity.com/paper/2502.16111/) (2025)
- [Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization](https://mlsanity.com/paper/2405.16681/) (2024)
- [PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving](https://mlsanity.com/paper/2507.07495/) (2025)
- [ToW: Thoughts of Words Improve Reasoning in Large Language Models](https://mlsanity.com/paper/2410.16235/) (2024)

## Built on by (most cited first, 9 of 9)

- [Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning](https://mlsanity.com/paper/2508.16949/) (2025)
- [Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning](https://mlsanity.com/paper/2511.12344/) (2025)
- [GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time](https://mlsanity.com/paper/2510.03777/) (2025)
- [FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments](https://mlsanity.com/paper/2604.25135/) (2026)
- [RECAP: Transparent Inference-Time Emotion Alignment for Medical Dialogue Systems](https://mlsanity.com/paper/2509.10746/) (2025)
- [Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution](https://mlsanity.com/paper/2604.03472/) (2026)
- [Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models](https://mlsanity.com/paper/2605.08472/) (2026)
- [OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation](https://mlsanity.com/paper/2510.03771/) (2025)
- [Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications](https://mlsanity.com/paper/2510.17764/) (2025)

## Abstract (arXiv)

Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning. While RL drives this self-improvement paradigm, a recent study (Gandhi et al., 2025) shows that RL alone does not truly instill these new reasoning abilities - it merely draws out behaviors already present in the base models. This raises a question: How can we train the models that don't exhibit such thinking behavior to develop it in the first place? To this end, we propose ThinkTuning, a GRPO-based interactive training approach where we augment the rollouts of a student model with the guidance from a teacher model. A simple idea from classroom practice inspires our method: a teacher poses a problem, lets the student try an answer, then gives corrective feedback -- enough to point the mind in the right direction and then show the solution. Each piece of feedback reshapes the student's thoughts, leading them to arrive at the correct solution. Similarly, we find that this type of implicit supervision through feedback from a teacher model of the same size improves the reasoning capabilities of the student model. In particular, on average, our method shows a 3.85% improvement over zero-shot baselines across benchmarks, and on MATH-500, AIME and GPQA-Diamond it shows 2.08%, 2.23% and 3.99% improvements over the vanilla-GRPO baseline. Source code is available at https://github.com/3rdAT/ThinkTuning.