# mR3: Multilingual Rubric-Agnostic Reward Reasoning Models

arXiv: https://arxiv.org/abs/2510.01146 · HTML: https://mlsanity.com/paper/2510.01146/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 70 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [Critique-out-Loud Reward Models](https://mlsanity.com/paper/2408.11791/) (2024)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization](https://mlsanity.com/paper/2402.16694/) (2024)
- [INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge](https://mlsanity.com/paper/2411.19799/) (2024)
- [JudgeLRM: Large Reasoning Models as a Judge](https://mlsanity.com/paper/2504.00050/) (2025)
- [HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages](https://mlsanity.com/paper/2505.11475/) (2025)
- [Self-Taught Evaluators](https://mlsanity.com/paper/2408.02666/) (2024)
- [PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages](https://mlsanity.com/paper/2504.04377/) (2025)
- [PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts](https://mlsanity.com/paper/2504.18428/) (2025)
- [M-RewardBench: Evaluating Reward Models in Multilingual Settings](https://mlsanity.com/paper/2410.15522/) (2024)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [Improving Reward Models with Synthetic Critiques](https://mlsanity.com/paper/2405.20850/) (2024)
- [M-Prometheus: A Suite of Open Multilingual LLM Judges](https://mlsanity.com/paper/2504.04953/) (2025)
- [Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://mlsanity.com/paper/2502.17407/) (2025)
- [MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models](https://mlsanity.com/paper/2410.17578/) (2024)

## Built on by (most cited first, 8 of 8)

- [Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization](https://mlsanity.com/paper/2604.07343/) (2026)
- [A Survey on Rubric-Guided Reinforcement Learning for Language Models](https://mlsanity.com/paper/2608.27505/) (2026)
- [Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck](https://mlsanity.com/paper/2603.10351/) (2026)
- [M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG](https://mlsanity.com/paper/2512.05959/) (2025)
- [Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation](https://mlsanity.com/paper/2604.11290/) (2026)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages](https://mlsanity.com/paper/2511.06497/) (2025)
- [UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms](https://mlsanity.com/paper/2609.05910/) (2026)

## Abstract (arXiv)

Evaluation using Large Language Model (LLM) judges has been widely adopted in English and shown to be effective for automatic evaluation. However, their performance does not generalize well to non-English settings, and it remains unclear what constitutes effective multilingual training for such judges. In this paper, we introduce mR3, a massively multilingual, rubric-agnostic reward reasoning model trained on 72 languages, achieving the broadest language coverage in reward modeling to date. We present a comprehensive study of data and curriculum selection for training to identify effective strategies and data sources for building high-quality reward models, including support for reasoning in the target language. Our approach attains state-of-the-art performance on multilingual reward model benchmarks, surpassing much larger models (i.e., GPT-OSS-120B) while being up to 9x smaller, and its effectiveness is further confirmed through extensive ablation studies. Finally, we demonstrate the effectiveness of mR3 in off-policy preference optimization and validate the quality of its reasoning traces and rubric-based evaluations through human studies with 20 annotators across 12 languages, where mR3 models' reasoning is preferred, including for extremely low-resource languages that are entirely unseen during training. Our models, data, and code are available as open source at https://github.com/rubricreward/mr3.