mlsanityData to 7 Oct 2026

Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

arXiv 2406.09136 ↗First posted 13 Jun 2024cs.CL cs.LGcited by 94

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at https://github.com/sail-sg/CPO.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 77 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

62 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation7 Nov
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Models18 Aug
  3. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework5 May
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 56

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
38 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2017Non-Autoregressive Neural Machine Translationcited by 457
  13. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  14. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  15. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  16. 2024Iterative Reasoning Preference Optimizationcited by 244
  17. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  18. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  19. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  20. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  21. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 166
  22. 2024Token-level Direct Preference Optimizationcited by 135
  23. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  24. 2023Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Methodcited by 97
  25. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  26. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  27. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  28. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  29. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  30. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizingcited by 49
  31. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  32. 2024Prompt Design and Engineering: Introduction and Advanced Methodscited by 45
  33. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  34. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedbackcited by 35
  35. 2022MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoningcited by 22
  36. 2018Explicit Interaction Model towards Text Classificationcited by 16
  37. 2024Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLMcited by 14
  38. 2023Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Modelscited by 13

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 66; 2026: 16.

94 papers.

Sort:

By area:

  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  2. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  3. 2024Distilling System 2 into System 1cited by 152
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  5. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  7. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  8. 2024Process Reward Model with Q-Value Rankingscited by 78
  9. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  10. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  11. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  12. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
38 more
  1. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  2. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  3. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 42
  4. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  7. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  8. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  9. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  10. 2025Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformercited by 24
  11. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  12. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  13. 2024Sample-Efficient Alignment for LLMscited by 16
  14. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  15. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  16. 2025HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinkingcited by 14
  17. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  18. 2025Credible Plan-Driven RAG Method for Multi-Hop Question Answeringcited by 12
  19. 2025CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learningcited by 12
  20. 2025CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidancecited by 11
  21. 2025Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generationcited by 11
  22. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  23. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancementcited by 10
  24. 2025Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Modelscited by 10
  25. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  26. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  27. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  28. 2025DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervisioncited by 9
  29. 2024Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoringcited by 8
  30. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  31. 2024Blind Spot Navigation in Large Language Model Reasoning with Thought Space Explorercited by 7
  32. 2025Structured Preference Optimization for Vision-Language Long-Horizon Task Planningcited by 7
  33. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  34. 2025LaRe: Latent Refocusing for Multimodal Reasoningcited by 7
  35. 2024Sparse Rewards Can Self-Train Dialogue Agentscited by 6
  36. 2024Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Datacited by 6
  37. 2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Valuescited by 6
  38. 2025Exploring Large Language Models for Word Games:Who is the Spy?cited by 6

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ChainPrune: Evaluating and Reducing Redundancy in Long Chain-of-Thought Reasoning
  2. 2025From Reasoning to Super-Intelligence: A Search-Theoretic Perspective
  3. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
  4. 2024On the Empirical Complexity of Reasoning and Planning in LLMs
  5. 2024Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning
  6. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs
  7. 2025Focused Chain-of-Thought: Efficient LLM Reasoning via Structured Input Information
  8. 2026Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency