mlsanityData to 7 Oct 2026

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning

arXiv 2410.02229 ↗First posted 3 Oct 2024cs.AI cs.CLcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have made significant progress in natural language understanding and generation, driven by scalable pretraining and advanced finetuning. However, enhancing reasoning abilities in LLMs, particularly via reinforcement learning from human feedback (RLHF), remains challenging due to the scarcity of high-quality preference data, which is labor-intensive to annotate and crucial for reward model (RM) finetuning. To alleviate this issue, we introduce CodePMP, a scalable preference model pretraining (PMP) pipeline that utilizes a large corpus of synthesized code-preference pairs from publicly available high-quality source code. CodePMP improves RM finetuning efficiency by pretraining preference models on large-scale synthesized code-preference pairs. We evaluate CodePMP on mathematical reasoning tasks (GSM8K, MATH) and logical reasoning tasks (ReClor, LogiQA2.0), consistently showing significant improvements in reasoning performance of LLMs and highlighting the importance of scalable preference model pretraining for efficient reward modeling.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 54 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

39 of 54 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  4. 2023Let's Verify Step by Step31 May
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
20 more
  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  6. 2023Scaling Data-Constrained Language Modelscited by 391
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  9. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  10. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  11. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  12. 2022Holistic Evaluation of Language Modelscited by 115
  13. 2024Critique-out-Loud Reward Modelscited by 105
  14. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  15. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  16. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58
  17. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  18. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  19. 2023Critique Ability of Large Language Modelscited by 20
  20. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Functioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 4.

10 papers.

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  3. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  4. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  5. 2026Self-Improving Pretraining: using post-trained models to pretrain better modelscited by 6
  6. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  7. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  8. 2025Libra: Large Chinese-based Safeguard for AI Contentcited by 2
  9. 2026Domain-Aware Scaling Laws Uncover Data Synergycited by 2
  10. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  2. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
  3. 2026PARM: Pipeline-Adapted Reward Model
  4. 2025Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning
  5. 2025Tiny Reward Models
  6. 2025Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
  7. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  8. 2024Plug-and-Play Training Framework for Preference Optimization