mlsanityData to 7 Oct 2026

Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch

arXiv 2502.17173 ↗First posted 24 Feb 2025cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are crucial for aligning large language models (LLMs) with human preferences. However, most RM research is centered on English and relies heavily on synthetic resources, which leads to limited and less reliable datasets and benchmarks for Chinese. To address this gap, we introduce CheemsBench, a fully human-annotated RM evaluation benchmark within Chinese contexts, and CheemsPreference, a large-scale and diverse preference dataset annotated through human-machine collaboration to support Chinese RM training. We systematically evaluate open-source discriminative and generative RMs on CheemsBench and observe significant limitations in their ability to capture human preferences in Chinese scenarios. Additionally, based on CheemsPreference, we construct an RM that achieves state-of-the-art performance on CheemsBench, demonstrating the necessity of human supervision in RM training. Our findings reveal that scaled AI-generated data struggles to fully capture human preferences, emphasizing the importance of high-quality human supervision in RM development.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 59 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

36 of 59 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  3. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings20 Oct
  4. 2024How to Evaluate Reward Models for RLHF18 Oct
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  6. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  7. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  10. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  11. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
3 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  7. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  8. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  9. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  10. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  11. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  12. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
9 more
  1. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  2. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  3. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  4. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  5. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  6. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  7. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  8. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21
  9. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Reward Models are Metrics in a Trench Coatcited by 2
  2. 2026Structuring the Space of Sociotechnical Alignmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling
  3. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  4. 2025RewardBench 2: Advancing Reward Model Evaluation
  5. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  6. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  7. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  8. 2024Transforming and Combining Rewards for Aligning Large Language Models