mlsanityData to 7 Oct 2026

M-RewardBench: Evaluating Reward Models in Multilingual Settings

arXiv 2410.15522 ↗First posted 20 Oct 2024cs.CL cs.AI cs.LGcited by 50

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) have driven the state-of-the-art performance of LLMs today by enabling the integration of human feedback into the language modeling process. However, RMs are primarily trained and evaluated in English, and their capabilities in multilingual settings remain largely understudied. In this work, we conduct a systematic evaluation of several reward models in multilingual settings. We first construct the first-of-its-kind multilingual RM evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances for 23 typologically diverse languages, that tests the chat, safety, reasoning, and translation capabilities of RMs. We then rigorously evaluate a wide range of reward models on M-RewardBench, offering fresh insights into their performance across diverse languages. We identify a significant gap in RMs' performances between English and non-English languages and show that RM preferences can change substantially from one language to another. We also present several findings on how different multilingual aspects impact RM performance. Specifically, we show that the performance of RMs is improved with improved translation quality. Similarly, we demonstrate that the models exhibit better performance for high-resource languages. We release M-RewardBench dataset and the codebase in this study to facilitate a better understanding of RM evaluation in multilingual settings.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 44 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

26 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  2. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  4. 2024Self-Rewarding Language Models18 Jan
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  12. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
8 more
  1. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  2. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  3. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  4. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 90
  5. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 68
  6. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  7. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  8. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 32; 2026: 12.

50 papers.

Sort:

By area:

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  2. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  3. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  6. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  7. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  8. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  9. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  10. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  11. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  12. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
38 more
  1. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  2. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  3. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  4. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  5. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  6. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  7. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  8. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  9. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 7
  10. 2024LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistenciescited by 6
  11. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
  12. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  13. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5
  14. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  15. 2025Making, not Taking, the Best of Ncited by 4
  16. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  17. 2025REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilitiescited by 3
  18. 2025Checklist Engineering Empowers Multilingual LLM Judgescited by 3
  19. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  20. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  21. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  22. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 2
  23. 2025Test-Time Scaling with Repeated Sampling Improves Multilingual Text Generationcited by 2
  24. 2025Reward Models are Metrics in a Trench Coatcited by 2
  25. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  26. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agentscited by 2
  27. 2025XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generationcited by 1
  28. 2025IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesiancited by 1
  29. 2026Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generationcited by 1
  30. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  31. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMscited by 1
  32. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 1
  33. 2025A Systematic Analysis of Base Model Choice for Reward Modelingcited by 0
  34. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  35. 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarkingcited by 0
  36. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluatorscited by 0
  37. 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdictcited by 0
  38. 2026CRISP: Cultural Reward Modeling for Implicit Situated Proprietycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  2. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  3. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  4. 2025BaseReward: A Strong Baseline for Multimodal Reward Model
  5. 2025Demystifying Multilingual Chain-of-Thought in Process Reward Modeling
  6. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignment
  7. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
  8. 2025RewardAnything: Generalizable Principle-Following Reward Models