mlsanityData to 7 Oct 2026

UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

arXiv 2609.05910 ↗First posted 5 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce MixReward, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose UniRRM, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 62 references that are in the map. First posted on arXiv 5 Sep 2026.

Builds on

45 of 62 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Models1 Oct
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  4. 2025RM-R1: Reward Modeling as Reasoning5 May
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2024A Survey on LLM-as-a-Judgecited by 1,384
  11. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  12. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
26 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  6. 2016FastText.zip: Compressing text classification modelscited by 331
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  8. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  11. 2024Generative Reward Modelscited by 148
  12. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  13. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  14. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  15. 2025Reinforcement Learning with Rubric Anchorscited by 108
  16. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  17. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  18. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  19. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  20. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 37
  21. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  22. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  23. 2025Guiding Retrieval using LLM-based Listwise Rankerscited by 16
  24. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  25. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  26. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  2. 2025Demystifying Multilingual Chain-of-Thought in Process Reward Modeling
  3. 2025URPO: A Unified Reward & Policy Optimization Framework for Large Language Models
  4. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  5. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model
  6. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
  7. 2025OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
  8. 2026GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings