mlsanityData to 14 Oct 2026

Self-Consistency Preference Optimization

arXiv 2411.04109 ↗First posted 6 Nov 2024cs.CL cs.AI cs.LGcited by 50

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

Self-alignment, whereby models learn to improve themselves without human annotation, is a rapidly growing research area. However, existing techniques often fail to improve complex reasoning tasks due to the difficulty of assigning correct rewards. An orthogonal approach that is known to improve correctness is self-consistency, a method applied at inference time based on multiple sampling in order to find the most consistent answer. In this work, we extend the self-consistency concept to help train models. We thus introduce self-consistency preference optimization (ScPO), which iteratively trains consistent answers to be preferred over inconsistent ones on unsupervised new problems. We show ScPO leads to large improvements over conventional reward model training on reasoning tasks such as GSM8K and MATH, closing the gap with supervised training with gold answers or preferences, and that combining ScPO with standard supervised learning improves results even further. On ZebraLogic, ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and Claude-3 Haiku.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 50 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

40 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Iterative Reasoning Preference Optimization30 Apr

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
27 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  5. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  9. 2024Self-Rewarding Language Modelscited by 681
  10. 2022Large Language Models Can Self-Improvecited by 664
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  12. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  13. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  14. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  15. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  16. 2022CodeT: Code Generation with Generated Testscited by 384
  17. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  18. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  19. 2019Addressing Failure Prediction by Learning Model Confidencecited by 186
  20. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  21. 2024Generative Reward Modelscited by 148
  22. 2022Natural Language to Code Translation with Executioncited by 124
  23. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  24. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  25. 2024Self-Taught Evaluatorscited by 72
  26. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 69
  27. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 30; 2026: 16.

50 papers.

Sort:

By area:

  1. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  2. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  3. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  4. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  6. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  7. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  8. 2025Self-rewarding correction for mathematical reasoningcited by 70
  9. 2025Can Large Reasoning Models Self-Train?cited by 68
  10. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  11. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 43
  12. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
38 more
  1. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  2. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  3. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 20
  4. 2026Recursive Synthesis for Long-Horizon Terminal Taskscited by 19
  5. 2025Learning to Generate Unit Tests for Automated Debuggingcited by 17
  6. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
  7. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  9. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 15
  10. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  11. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 13
  12. 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
  13. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  14. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 12
  15. 2025Unsupervised Elicitation of Language Modelscited by 11
  16. 2026Entropy-Preserving Reinforcement Learningcited by 11
  17. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 10
  18. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Modelscited by 9
  19. 2026Tool Verification for Test-Time Reinforcement Learningcited by 9
  20. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  21. 2025Self-Training Large Language Models with Confident Reasoningcited by 8
  22. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 7
  23. 2025PRInTS: Reward Modeling for Long-Horizon Information Seekingcited by 5
  24. 2026Consensus as Privileged Context for Label-Free Self-Distillationcited by 5
  25. 2025Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Samplingcited by 4
  26. 2026DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuningcited by 4
  27. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  28. 2025Probability-Consistent Preference Optimization for Enhanced LLM Reasoningcited by 2
  29. 2026Provable and Practical In-Context Policy Optimization for Self-Improvementcited by 2
  30. 2025Atomic Consistency Preference Optimization for Long-Form Question Answeringcited by 1
  31. 2025IRIS: Intrinsic Reward Image Synthesiscited by 1
  32. 2026TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordinationcited by 1
  33. 2026J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Datacited by 1
  34. 2025The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Trainingcited by 0
  35. 2026Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMscited by 0
  36. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
  37. 2026Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoningcited by 0
  38. 2026Improving scalable oversight with co-trained monitorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning
  2. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  3. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  4. 2025Semi-Supervised Preference Optimization with Limited Feedback
  5. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  6. 2025SGPO: Self-Generated Preference Optimization based on Self-Improver
  7. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning
  8. 2026Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning