mlsanityData to 7 Oct 2026

Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

arXiv 2410.18451 ↗First posted 24 Oct 2024cs.AI cs.CLcited by 322

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

In this report, we introduce a collection of methods to enhance reward modeling for LLMs, focusing specifically on data-centric techniques. We propose effective data selection and filtering strategies for curating high-quality open-source preference datasets, culminating in the Skywork-Reward data collection, which contains only 80K preference pairs -- significantly smaller than existing datasets. Using this curated dataset, we developed the Skywork-Reward model series -- Skywork-Reward-Gemma-27B and Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top position on the RewardBench leaderboard. Notably, our techniques and datasets have directly enhanced the performance of many top-ranked models on RewardBench, highlighting the practical impact of our contributions in real-world preference learning applications.

Key facts. Cited by 322 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 50 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

38 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
22 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  4. 2024InternLM2 Technical Reportcited by 391
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  7. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  10. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  11. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  12. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  13. 2024Nemotron-4 340B Technical Reportcited by 143
  14. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  15. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  16. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  17. 2024Stable LM 2 1.6B Technical Reportcited by 86
  18. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  19. 2024Self-Taught Evaluatorscited by 70
  20. 2024Direct Judgement Preference Optimizationcited by 33
  21. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  22. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 204; 2026: 106.

322 papers.

Sort:

By area:

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  2. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  5. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  6. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  7. 2024Free Process Rewards without Process Labelscited by 162
  8. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  9. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  10. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  11. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  12. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
38 more
  1. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  2. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  3. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  4. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  5. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  6. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  7. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  8. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  9. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  10. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  11. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  12. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  13. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  14. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  15. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  16. 2025ExGRPO: Learning to Reason from Experiencecited by 49
  17. 2025Reward Reasoning Modelcited by 47
  18. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  19. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  20. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  21. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  22. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 40
  23. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  24. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  25. 2025Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoningcited by 36
  26. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  27. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  28. 2024Evaluating Language Models as Synthetic Data Generatorscited by 32
  29. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewardingcited by 32
  30. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  31. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  32. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 31
  33. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  34. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  35. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  36. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 26
  37. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  38. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25

50 most cited of 322 All 322 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  2. 2026Scaling Reward Modeling without Human Supervision
  3. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
  4. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
  5. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning
  6. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2023Tool-Augmented Reward Modeling