mlsanityData to 7 Oct 2026

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

arXiv 2410.21545 ↗First posted 28 Oct 2024cs.CLcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward modeling in large language models is susceptible to reward hacking, causing models to latch onto superficial features such as the tendency to generate lists or unnecessarily long responses. In reinforcement learning from human feedback (RLHF) and more generally during post-training flawed reward signals often lead to outputs that optimize for these spurious correlates instead of genuine quality or correctness. We propose Context-Aware Reward Modeling (CARMO), a novel approach that first generates dynamic, context-relevant criteria to ground the reward model before producing reward scores. Unlike prior methods that rely on static rubrics, CARMO leverages large language models (LLMs) to adaptively create evaluation criteria such as logical consistency, clarity, and depth tailored to the user query. Our theoretical analysis shows that such criteria generation can mitigate reward hacking. We further demonstrate that CARMO can be distilled into smaller models, reducing the computational cost of alignment. We establish a new state-of-the-art performance in zero-shot settings for generative models, achieving a 2.1% improvement on Reward Bench. Furthermore, alignment performed on the CARMO-curated preference dataset achieves 22.5% and 21.1% LC-WR and WR, respectively, on Mistral-Base (7B).

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 33 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

21 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
3 more
  1. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  2. 2022Evaluating Human-Language Model Interactioncited by 90
  3. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 17.

21 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  3. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  4. 2025Robust Reward Modeling via Causal Rubricscited by 18
  5. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  6. 2025MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answeringcited by 8
  7. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  8. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 7
  9. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  10. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  11. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  12. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3
9 more
  1. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 3
  2. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluationcited by 2
  3. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 1
  4. 2026Task-Adaptive Rubrics for GUI Reward Modelingcited by 1
  5. 2026Tape: A Cellular Automata Benchmark for Evaluating Rule-Shift Generalization in Reinforcement Learningcited by 0
  6. 2026RASP-Tuner: Retrieval-Augmented Soft Prompts for Context-Aware Black-Box Optimization in Non-Stationary Environmentscited by 0
  7. 2026Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?cited by 0
  8. 2026Test-Time Scaling via Error Localizationcited by 0
  9. 2026RLVR2: Reinforcement Learning with Verifiable Rubric-based Rankingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  2. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
  3. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generation
  4. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  5. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
  6. 2026RMB: Reward Model Boosting Mitigates Reward Hacking
  7. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  8. 2025Reward Models are Metrics in a Trench Coat