# Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints

arXiv: https://arxiv.org/abs/2511.16139 · HTML: https://mlsanity.com/paper/2511.16139/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains](https://mlsanity.com/paper/2503.23829/) (2025)
- [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025)
- [InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model](https://mlsanity.com/paper/2501.12368/) (2025)
- [Baichuan-M2: Scaling Medical Capability with Large Verifier System](https://mlsanity.com/paper/2509.02208/) (2025)
- [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025)
- [Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences](https://mlsanity.com/paper/2403.01857/) (2024)
- [A Survey on Progress in LLM Alignment from the Perspective of Reward Design](https://mlsanity.com/paper/2505.02666/) (2025)
- [Pre-Trained Policy Discriminators are General Reward Models](https://mlsanity.com/paper/2507.05197/) (2025)
- [Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards](https://mlsanity.com/paper/2505.02686/) (2025)
- [Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning](https://mlsanity.com/paper/2509.25534/) (2025)
- [Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions](https://mlsanity.com/paper/2404.13307/) (2024)

## Built on by (most cited first, 4 of 4)

- [Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning](https://mlsanity.com/paper/2606.03361/) (2026)
- [ClinAlign: Scaling Healthcare Alignment from Clinician Preference](https://mlsanity.com/paper/2602.09653/) (2026)
- [Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigm](https://mlsanity.com/paper/2602.11661/) (2026)
- [ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning](https://mlsanity.com/paper/2603.06065/) (2026)

## Abstract (arXiv)

The integration of large language models (LLMs) into medical practice offers transformative potential, yet their real-world clinical applicability remains constrained by critical alignment issues: (1) a misalignment between static evaluation benchmarks and the dynamic cognitive demands of clinical practice, (2) challenges in adapting to continuously evolving, multi-source medical standards, and (3) the limited capacity of conventional reward models to reflect nuanced, multi-dimensional medical quality criteria. To overcome these limitations, we introduce MR-RML (Multidimensional Rubric-oriented Reward Model Learning) with GPRC (Geometric Projection Reference Constraints)-a novel alignment framework that structured medical standards into a multi-perspective matrix to guide both data generation and model optimization. Our approach introduces three key innovations: (1) a medical standard system that embeds domain-specific guidelines throughout the training pipeline; (2) an independent multi-dimensional reward model that decomposes evaluation criteria, transitioning from rule-based or LLM-based scoring to internalized reward modeling for better evaluation performance; and (3) geometric projection reference constraints that translate clinical cognitive logic into mathematical regularization, aligning scoring gradients with clinical reasoning and facilitating training with synthetically generated data. Extensive evaluations on the authoritative medical benchmark Healthbench demonstrate that our method significantly boosts the performance of the base Qwen-32B model, with improvements of 45% on the full subset and 85% on the hard subset. It achieves state-of-the-art results among open-source LLMs, scoring 62.7 (full) and 44.7 (hard), while also surpassing the majority of closed-source models.