mlsanityData to 14 Oct 2026

Checklists Are Better Than Reward Models For Aligning Language Models

arXiv 2507.18624 ↗First posted 24 Jul 2025cs.CLcited by 128

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this -- typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item - using both AI judges and specialized verifier programs - then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods applied to a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks -- RLCF is the only method to improve performance on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. These results establish checklist feedback as a key tool for improving language models' support of queries that express a multitude of needs.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 55 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

47 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  2. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  4. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  6. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 38

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
26 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  7. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  8. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  10. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  11. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  12. 20242 OLMo 2 Furiouscited by 340
  13. 2025Generalizing Verifiable Instruction Followingcited by 194
  14. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  15. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  16. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  17. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  18. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  19. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  20. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  21. 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 38
  22. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  23. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  24. 2024Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Modelscited by 27
  25. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  26. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 105.

128 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  3. 2025Reinforcement Learning with Rubric Anchorscited by 113
  4. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  5. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  6. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  7. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  8. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  9. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  10. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  11. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 30
  12. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
38 more
  1. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  2. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 26
  3. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
  4. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  5. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 21
  6. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  7. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  8. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  9. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
  10. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
  11. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  12. 2026Out of Sight, Out of Mind? Evaluating State Evolution in Video World Modelscited by 13
  13. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 13
  14. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12
  15. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  16. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  17. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  18. 2025From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational Systemcited by 9
  19. 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
  20. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  21. 2026Training AI Scientists to Replicate Researchcited by 9
  22. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 8
  23. 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Taskscited by 7
  24. 2025Making, not Taking, the Best of Ncited by 6
  25. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  26. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 6
  27. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  28. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
  29. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  30. 2025Evaluating Legal Reasoning Traces with Legal Issue Tree Rubricscited by 5
  31. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 5
  32. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5
  33. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judgecited by 5
  34. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  35. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  36. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
  37. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 5
  38. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 5

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  2. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  3. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  5. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  6. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  7. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective