Data to 7 Oct 2026

Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy

arXiv 2507.01352 ↗First posted 2 Jul 2025cs.CL cs.AI cs.LGcited by 204

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Despite the critical role of reward models (RMs) in Reinforcement Learning from Human Feedback (RLHF), current state-of-the-art open RMs perform poorly on most existing evaluation benchmarks, failing to capture nuanced human preferences. We hypothesize that this brittleness stems primarily from limitations in preference datasets, which are often narrowly scoped, synthetically labeled, or lack rigorous quality control. To address these challenges, we present SynPref-40M, a large-scale preference dataset comprising 40 million preference pairs. To enable data curation at scale, we design a human-AI synergistic two-stage pipeline that leverages the complementary strengths of human annotation quality and AI scalability. In this pipeline, humans provide verified annotations, while LLMs perform automatic curation based on human guidance. Training on this preference mixture, we introduce Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B parameters, trained on a carefully curated subset of 26 million preference pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile across a wide range of capabilities, including alignment with human preferences, objective correctness, safety, resistance to stylistic biases, and best-of-N scaling. These reward models achieve state-of-the-art performance across seven major reward model benchmarks, outperform generative reward models, and demonstrate strong downstream performance. Ablation studies confirm that effectiveness stems not only from data scale but also from high-quality curation. The Skywork-Reward-V2 series represents substantial progress in open reward models, demonstrating how human-AI curation synergy can unlock significantly higher data quality.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 82 of its 116 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

82 of 116 references.

Closest ancestors · 14

Same direction, newest first

2 more

2022Constitutional AI: Harmlessness from AI Feedback15 Dec
2017Proximal Policy Optimization Algorithms20 Jul

Other references · 68

2024The Llama 3 Herd of Modelscited by 14,192
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2025Qwen3 Technical Reportcited by 8,195
2021Evaluating Large Language Models Trained on Codecited by 8,053
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
2024Qwen2.5 Technical Reportcited by 4,713
2024GPT-4o System Cardcited by 4,467
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2021Program Synthesis with Large Language Modelscited by 3,509
2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928

38 more

2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
2025Gemma 3 Technical Reportcited by 1,686
2024OpenAI o1 System Cardcited by 1,220
2024Qwen2.5-Coder Technical Reportcited by 1,151
2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
2024DeepSeek-V3 Technical Reportcited by 724
2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
20242 OLMo 2 Furiouscited by 302
2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
2024Direct Language Model Alignment from Online AI Feedbackcited by 257
2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
2022Is GPT-3 a Good Data Annotator?cited by 208
2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
2025Skywork Open Reasoner 1 Technical Reportcited by 161
2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
2024How to Evaluate Reward Models for RLHFcited by 84
2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
2024Self-Taught Evaluatorscited by 70
2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
2024Quantile Regression for Distributional Reward Models in RLHFcited by 59

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 53; 2026: 151.

204 papers.

Sort:

By area:

2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
2025Language Self-Play For Data-Free Trainingcited by 42
2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
2026On the Position Bias of On-Policy Distillationcited by 23
2025Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learningcited by 22
2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 19
2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18

38 more

2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
2025Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuitscited by 12
2026Real-Time Aligned Reward Model beyond Semanticscited by 12
2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 12
2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 12
2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 8
2025DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimizationcited by 7
2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
2025Multiplayer Nash Preference Optimizationcited by 7
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learningcited by 7
2025Majority of the Bests: Improving Best-of-N via Bootstrappingcited by 6
2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
2026Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systemscited by 6
2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
2026ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generationcited by 5
2026PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Modelscited by 5
2026MAS-ProVe: Understanding the Process Verification of Multi-Agent Systemscited by 5
2026GOPO: Policy Optimization using Ranked Rewardscited by 5
2026Scaling Agentic Verifier for Competitive Codingcited by 5
2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
2026MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucinationcited by 5
2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 5
2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
2025RTTC: Reward-Guided Collaborative Test-Time Computecited by 4
2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 4
2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way