mlsanityData to 7 Oct 2026

Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring

arXiv 2605.00754 ↗First posted 1 May 2026cs.SE cs.LGcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 140 of its 316 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

140 of 316 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systems8 Sep
  2. 2025PurpCode: Reasoning for Safer Code Generation25 Jul
  3. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset27 May
  4. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning19 Apr
  5. 2025R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation7 Apr
  6. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models23 Feb
  7. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation13 Feb

Other references · 133

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
38 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  6. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  7. 2024InternLM2 Technical Reportcited by 391
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  9. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  11. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  12. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  13. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  14. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  15. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  16. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  17. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  18. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  19. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  20. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  21. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  22. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  23. 2024Generative Reward Modelscited by 148
  24. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  25. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  26. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  27. 2025Reinforcement Learning with Rubric Anchorscited by 108
  28. 2024Critique-out-Loud Reward Modelscited by 105
  29. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  30. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  31. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  32. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  33. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  34. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 75
  35. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  36. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  37. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  38. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70

50 most cited of 133 All 133 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewards
  2. 2025Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs
  3. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  4. 2026PARM: Pipeline-Adapted Reward Model
  5. 2026FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation
  6. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Models
  7. 2026Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation
  8. 2025Dynamic Scaling of Unit Tests for Code Reward Modeling