# PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling

arXiv: https://arxiv.org/abs/2510.24235 · HTML: https://mlsanity.com/paper/2510.24235/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 28 of 28)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024)
- [RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style](https://mlsanity.com/paper/2410.16184/) (2024)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [Checklists Are Better Than Reward Models For Aligning Language Models](https://mlsanity.com/paper/2507.18624/) (2025)
- [JudgeLRM: Large Reasoning Models as a Judge](https://mlsanity.com/paper/2504.00050/) (2025)
- [Improving Reward Models with Synthetic Critiques](https://mlsanity.com/paper/2405.20850/) (2024)
- [RewardAnything: Generalizable Principle-Following Reward Models](https://mlsanity.com/paper/2506.03637/) (2025)
- [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025)
- [When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoning](https://mlsanity.com/paper/2505.15400/) (2025)
- [GRAM: A Generative Foundation Reward Model for Reward Generalization](https://mlsanity.com/paper/2506.14175/) (2025)
- [R3: Robust Rubric-Agnostic Reward Models](https://mlsanity.com/paper/2505.13388/) (2025)
- [A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization](https://mlsanity.com/paper/2504.04950/) (2025)
- [Reasoner for Real-World Event Detection: Scaling Reinforcement Learning via Adaptive Perplexity-Aware Sampling Strategy](https://mlsanity.com/paper/2507.01327/) (2025)

## Built on by (most cited first, 6 of 6)

- [Exploring Reasoning Reward Model for Agents](https://mlsanity.com/paper/2601.22154/) (2026)
- [PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges](https://mlsanity.com/paper/2605.30803/) (2026)
- [Reward Modeling for Scientific Writing Evaluation](https://mlsanity.com/paper/2601.11374/) (2026)
- [Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents](https://mlsanity.com/paper/2603.01481/) (2026)
- [Learning Latent Reasoning Traces for Scalar Reward Models End-to-End](https://mlsanity.com/paper/2607.29185/) (2026)
- [Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges](https://mlsanity.com/paper/2610.18766/) (2026)

## Abstract (arXiv)

Reward models (RMs) are central to reinforcement learning from human feedback (RLHF), providing the critical supervision signals that align large language models (LLMs) with human preferences. Generative reward models (GRMs) provide greater interpretability than traditional scalar RMs, but they come with a critical trade-off: pairwise methods are hindered by a training-inference mismatch, while pointwise methods require expensive absolute annotations. To bridge this gap, we propose the Preference-aware Task-adaptive Reward Model (PaTaRM). Unlike prior approaches, PaTaRM enables robust pointwise training using readily available pairwise data via a novel Preference-Aware Reward (PAR) mechanism, eliminating the need for explicit rating labels. Furthermore, it incorporates a Task-Adaptive Rubric system that dynamically generates instance-specific criteria for precise evaluation. Extensive experiments demonstrate that PATRM achieves a 8.7% average improvement on RewardBench and RMBench across Qwen3-8B/14B models. Crucially, it boosts downstream RLHF performance by an average relative improvement of 13.6% across IFEval and InFoBench, validating its effectiveness for policy alignment. Our code is available at https://github.com/JaneEyre0530/PaTaRM.