Data to 14 Oct 2026

Quantile Regression for Distributional Reward Models in RLHF

arXiv 2409.10164 ↗First posted 16 Sep 2024cs.LG cs.AI cs.CLcited by 60

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has become a key method for aligning large language models (LLMs) with human preferences through the use of reward models. However, traditional reward models typically generate point estimates, which oversimplify the diversity and complexity of human values and preferences. In this paper, we introduce Quantile Reward Models (QRMs), a novel approach to reward modeling that learns a distribution over rewards instead of a single scalar value. Our method uses quantile regression to estimate a full, potentially multimodal distribution over preferences, providing a more powerful and nuanced representation of preferences. This distributional approach can better capture the diversity of human values, addresses label noise, and accommodates conflicting preferences by modeling them as distinct modes in the distribution. Our experimental results show that QRM outperforms comparable traditional point-estimate models on RewardBench. Furthermore, we demonstrate that the additional information provided by the distributional estimates can be utilized in downstream applications, such as risk-aware reinforcement learning, resulting in LLM policies that generate fewer extremely negative responses. Our code and model are released at https://github.com/Nicolinho/QRM.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 36 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

26 of 36 references.

Closest ancestors · 8

Same direction, newest first

Other references · 18

2023GPT-4 Technical Reportcited by 18,396
2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2023Let's Verify Step by Stepcited by 3,789
2020Learning to summarize from human feedbackcited by 2,588
2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
2017A Distributional Perspective on Reinforcement Learningcited by 765
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
2017Distributional Reinforcement Learning with Quantile Regressioncited by 474

6 more

2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 57
2024Aligning Crowd Feedback via Distributional Preference Reward Modelingcited by 28
2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 41; 2026: 16.

60 papers.

Sort:

By area:

2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 51
2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 32
2025Language Model Personalization via Reward Factorizationcited by 31
2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 24
2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 17

38 more

2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalizationcited by 15
2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
2025Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMscited by 10
2025Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?cited by 9
2025Capturing Individual Human Preferences with Reward Featurescited by 8
2025Data-adaptive Safety Rules for Training Reward Modelscited by 6
2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generationcited by 5
2026GOPO: Policy Optimization using Ranked Rewardscited by 5
2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
2025Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarkingcited by 4
2025Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judgecited by 4
2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
2025Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstructioncited by 3
2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
2025CTTS: Collective Test-Time Scalingcited by 3
2026Reward Models Inherit Value Biases from Pretrainingcited by 3
2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
2024Holistic Utility Preference Learning for Listwise Alignmentcited by 2
2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
2025Multi-Domain Explainability of Preferencescited by 2
2025Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignmentcited by 2
2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
2026SteerRM: Debiasing Reward Models via Sparse Autoencoderscited by 2
2025Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferencescited by 1
2025Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generationcited by 1
2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregationcited by 1
2025DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Trainingcited by 1
2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
2026Learning Ordinal Probabilistic Reward from Preferencescited by 1
2026Text-to-Distribution Prediction with Quantile Tokens and Neighbor Contextcited by 1
2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
2026StoryAlign: Evaluating and Training Reward Models for Story Generationcited by 1
2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
2025A Systematic Analysis of Base Model Choice for Reward Modelingcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way