mlsanityData to 14 Oct 2026

GRAM: A Generative Foundation Reward Model for Reward Generalization

arXiv 2506.14175 ↗First posted 17 Jun 2025cs.CL cs.AIcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference data. In this paper, we explore methods that train reward models using both unlabeled and labeled data. Building on the generative models in LLMs, we develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning. We also show that by using label smoothing, we are in fact optimizing a regularized pairwise ranking loss. This result, in turn, provides a new view of training reward models, which links generative models and discriminative models under the same class of training objectives. The outcome of these techniques is a foundation reward model, which can be applied to a wide range of tasks with little or no further fine-tuning effort. Extensive experiments show that this model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning, achieving significant performance improvements over several strong baseline models.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 53 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

43 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024Generative Reward Models2 Oct
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  4. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  6. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Everyone Deserves A Reward: Learning Customized Human Preferences6 Sep
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
18 more
  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  4. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  5. 2020Does label smoothing mitigate label noise?cited by 204
  6. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  7. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  8. 2025Reward Reasoning Modelcited by 47
  9. 2019Fine-grained Sentiment Classification using BERTcited by 38
  10. 2022A Survey on Aspect-Based Sentiment Classificationcited by 18
  11. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  12. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Datacited by 15
  13. 2025Foundations of Large Language Modelscited by 14
  14. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selectioncited by 12
  15. 2023Introduction to Transformers: an NLP Perspectivecited by 10
  16. 2024Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generationcited by 10
  17. 2024Hybrid Alignment Training for Large Language Modelscited by 7
  18. 2024NDP: Next Distribution Prediction as a More Broad Targetcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 11.

25 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  3. 2025Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Modelscited by 10
  4. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  5. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  6. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  7. 2025StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Modelscited by 6
  8. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 6
  9. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 5
  10. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  11. 2025HEAL: A Hypothesis-Based Preference-Aware Analysis Frameworkcited by 4
  12. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 4
13 more
  1. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  2. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  3. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  4. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 2
  5. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillationcited by 2
  6. 2026SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Modelscited by 2
  7. 2026AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizingcited by 2
  8. 2025Voting with the Graph: Stable RLAIF via Topological Consistency Maximizationcited by 1
  9. 2026ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Frameworkcited by 1
  10. 2026Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learningcited by 1
  11. 2025Bradley-Terry Policy Optimization for Generative Preference Modelingcited by 0
  12. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learningcited by 0
  13. 2026SQUARE: Structured Quantum Representation Adapters as Compact Quadratic Feature Maps for Frozen Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  2. 2019Fine-Tuning Language Models from Human Preferences
  3. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  4. 2024Semi-Supervised Reward Modeling via Iterative Self-Training
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  6. 2024Improving Reward Models with Synthetic Critiques
  7. 2025Generalist Reward Models: Found Inside Large Language Models
  8. 2026Reward Modeling from Natural Language Human Feedback