mlsanityData to 14 Oct 2026

Preference Optimization for Reasoning with Pseudo Feedback

arXiv 2411.16345 ↗First posted 25 Nov 2024cs.CLcited by 39

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

Preference optimization techniques, such as Direct Preference Optimization (DPO), are frequently employed to enhance the reasoning capabilities of large language models (LLMs) in domains like mathematical reasoning and coding, typically following supervised fine-tuning. These methods rely on high-quality labels for reasoning tasks to generate preference pairs; however, the availability of reasoning datasets with human-verified labels is limited. In this study, we introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions to reason problems as an evaluation against associated test cases. We explore two forms of pseudo feedback based on test cases: one generated by frontier LLMs and the other by extending self-consistency to multi-test-case. We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks. Specifically, using Mathstral-7B as our base model, we improve MATH results from 58.3 to 68.6, surpassing both NuminaMath-72B and GPT-4-Turbo-1106-preview. In GSM8K and College Math, our scores increase from 85.6 to 90.3 and from 34.3 to 42.3, respectively. Building on Deepseek-coder-7B-v1.5, we achieve a score of 24.6 on LiveCodeBench (from 21.1), surpassing Claude-3-Haiku.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 72 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

30 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  2. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Math16 Feb
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2024Self-Rewarding Language Modelscited by 681
15 more
  1. 2022Large Language Models Can Self-Improvecited by 664
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  4. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  5. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  6. 2023Self-Consuming Generative Models Go MADcited by 250
  7. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  8. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  9. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  10. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  11. 2024Prover-Verifier Games improve legibility of LLM outputscited by 85
  12. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizingcited by 49
  13. 2024Weak-to-Strong Reasoningcited by 33
  14. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  15. 2023Enabling Language Models to Implicitly Learn Self-Improvementcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 29; 2026: 9.

39 papers.

Sort:

By area:

  1. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  2. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 128
  3. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  4. 2025S*: Test Time Scaling for Code Generationcited by 98
  5. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 93
  6. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 90
  7. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  8. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  9. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 63
  10. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  11. 2025Scaling Laws of Synthetic Data for Language Modelscited by 46
  12. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
27 more
  1. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
  2. 2025GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chainscited by 14
  3. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  4. 2025AdsQA: Towards Advertisement Video Understandingcited by 13
  5. 2025Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answerscited by 11
  6. 2025Unsupervised Elicitation of Language Modelscited by 11
  7. 2025Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposurecited by 10
  8. 2025Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQLcited by 9
  9. 2025GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoningcited by 8
  10. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 7
  11. 2025Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedbackcited by 6
  12. 2025Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuningcited by 4
  13. 2025How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspectivecited by 4
  14. 2025QueST: Incentivizing LLMs to Generate Difficult Problemscited by 4
  15. 2025SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfacescited by 4
  16. 2024StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputscited by 3
  17. 2026CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Modelscited by 3
  18. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  19. 2026AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systemscited by 3
  20. 2026Locatability-Guided Adaptive Reasoning for Image Geo-Localization with Vision-Language Modelscited by 2
  21. 2025Collaborative Agents for Automated Program Repair in Rubycited by 1
  22. 2026SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignmentcited by 1
  23. 2026Autoregressive Direct Preference Optimizationcited by 1
  24. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensuscited by 1
  25. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  26. 2026GeoReward: Mitigating Contextual Variable Overestimation in Vision-Language Models for Cross-Market Preference Predictioncited by 0
  27. 2026Anchors Enhance Controllability and Reasoning of Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  2. 2025Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
  3. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning
  4. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  5. 2024Step-level Value Preference Optimization for Mathematical Reasoning
  6. 2024Plug-and-Play Training Framework for Preference Optimization
  7. 2025MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning
  8. 2024Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editing