mlsanityData to 14 Oct 2026

Thinking Preference Optimization

arXiv 2502.13173 ↗First posted 17 Feb 2025cs.LG cs.AIcited by 8

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Supervised Fine-Tuning (SFT) has been a go-to and effective method for enhancing long chain-of-thought (CoT) reasoning in relatively small LLMs by fine-tuning them with long CoT responses from larger LLMs. To continually improve reasoning abilities, we can either collect new high-quality long CoT reasoning SFT data or repeatedly train on existing SFT datasets. However, acquiring new long CoT SFT data is costly and limited, while repeated training often results in a performance plateau or decline. To further boost the performance with the SFT data, we propose Thinking Preference Optimization (ThinkPO), a simple yet effective post-SFT method that enhances long CoT reasoning without requiring new long CoT responses. Instead, ThinkPO utilizes readily available or easily obtainable short CoT reasoning responses as rejected answers and long CoT responses as chosen answers for the same question. It then applies direct preference optimization to encourage the model to favor longer reasoning outputs. Experiments show that ThinkPO further improves the reasoning performance of SFT-ed models, e.g. it increases math reasoning accuracy of SFT-ed models by 8.6% and output length by 25.9%. Notably, ThinkPO is capable of continually boosting the performance of the publicly distilled SFT model, e.g., increasing the official DeepSeek-R1-Distill-Qwen-7B's performance on MATH500 from 87.4% to 91.2%.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 50 of its 59 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

50 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025s1: Simple test-time scaling31 Jan
  2. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2024OpenAI o1 System Cardcited by 1,244
34 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  5. 2024Qwen2-Audio Technical Reportcited by 560
  6. 2024Large Language Models: A Surveycited by 525
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  8. 2025LIMO: Less is More for Reasoningcited by 498
  9. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  11. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  12. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  13. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  14. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  15. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  16. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  17. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  18. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  19. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 102
  20. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  21. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  22. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  23. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  24. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  25. 2024Process Reward Model with Q-Value Rankingscited by 80
  26. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  27. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 39
  28. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  29. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  30. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  31. 2025o3-mini vs DeepSeek-R1: Which One is Safer?cited by 24
  32. 2022A2C is a special case of PPOcited by 16
  33. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  34. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithmscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Learning to Reason under Off-Policy Guidancecited by 281
  3. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 29
  4. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  5. 2025CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generationcited by 19
  6. 2025AutoL2S: Auto Long-Short Reasoning for Efficient Large Language Modelscited by 12
  7. 2026Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decodingcited by 0
  8. 2026Retrospective Distillation Attribution via Normalized Response Similaritycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ReFT: Reasoning with Reinforced Fine-Tuning
  2. 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models
  3. 2025QFFT, Question-Free Fine-Tuning for Adaptive Reasoning
  4. 2025Rethinking Supervised Fine-Tuning: Emphasizing Key Answer Tokens for Improved LLM Accuracy
  5. 2024Iterative Reasoning Preference Optimization
  6. 2025Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs
  7. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
  8. 2026Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Training