mlsanityData to 14 Oct 2026

WorldPM: Scaling Human Preference Modeling

arXiv 2505.10527 ↗First posted 15 May 2025cs.CLcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling. We propose World Preference Modeling (WorldPM) to emphasize this scaling potential, where World Preference embodies a unified representation of human preferences. In this paper, we collect preference data from public forums covering diverse user communities, and conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters. We observe distinct patterns across different evaluation metrics: (1) Adversarial metrics (ability to identify deceptive features) consistently scale up with increased training data and base model size; (2) Objective metrics (objective knowledge with well-defined answers) show emergent behavior in larger language models, highlighting WorldPM's scalability potential; (3) Subjective metrics (subjective preferences from a limited number of humans or AI) do not demonstrate scaling trends. Further experiments validate the effectiveness of WorldPM as a foundation for preference fine-tuning. Through evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly improves the generalization performance across human preference datasets of varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5% on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we observe significant improvements on both in-house and public evaluation sets, with notable gains of 4% to 8% in our in-house evaluations.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 55 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

38 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking23 Sep
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
22 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  11. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  12. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  13. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  14. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  15. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  16. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  17. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  18. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  19. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  20. 2024How to Evaluate Reward Models for RLHFcited by 88
  21. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  22. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  2. 2025Qwen3Guard Technical Reportcited by 155
  3. 2025RewardDance: Reward Scaling in Visual Generationcited by 68
  4. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  5. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  6. 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
  7. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9
  8. 2026Leveraging Verifier-Based Reinforcement Learning in Image Editingcited by 9
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  10. 2026RewardHarness: Self-Evolving Agentic Post-Trainingcited by 5
  11. 2026AesRM: Improving Video Aesthetics with Expert-Level Feedbackcited by 3
  12. 2026RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Constructioncited by 2
1 more
  1. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Human Preference Learning for Large Language Models
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024Dissecting Human and LLM Preferences
  4. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  5. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  6. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  7. 2025HPS: Hard Preference Sampling for Human Preference Alignment
  8. 2023Compositional preference models for aligning LMs