mlsanityData to 7 Oct 2026

Rule Based Rewards for Language Model Safety

arXiv 2411.01111 ↗First posted 2 Nov 2024cs.AIcited by 106

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental. Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental) along with a LLM grader. In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating. We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 28 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

24 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
10 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  7. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  8. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  9. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  10. 2023Specific versus General Principles for Constitutional AIcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 61; 2026: 39.

106 papers.

Sort:

By area:

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  2. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  3. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  5. 2025Reinforcement Learning with Rubric Anchorscited by 108
  6. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  7. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  8. 2025One Token to Fool LLM-as-a-Judgecited by 56
  9. 2025LongCat-Flash Technical Reportcited by 54
  10. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  11. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  12. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
38 more
  1. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  2. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  3. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learningcited by 32
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  5. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  6. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  7. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  8. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 28
  9. 2025SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuningcited by 27
  10. 2025OpenAI's Approach to External Red Teaming for AI Models and Systemscited by 26
  11. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  12. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  13. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Modelscited by 22
  14. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 22
  15. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judgescited by 19
  16. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  17. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  18. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  19. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  20. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  21. 2025Safety Reasoning with Guidelinescited by 14
  22. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  23. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 14
  24. 2025Perception in Reflectioncited by 13
  25. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Modelscited by 13
  26. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoningcited by 12
  27. 2025Reinforcement Learning on Pre-Training Datacited by 12
  28. 2024Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMscited by 11
  29. 2025Stress-Testing Model Specs Reveals Character Differences among Language Modelscited by 10
  30. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
  31. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 10
  32. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 10
  33. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agentscited by 9
  34. 2025Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMscited by 8
  35. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  36. 2025Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?cited by 8
  37. 2026Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Securitycited by 8
  38. 2025The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMscited by 7

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024Evaluating Defences against Unsafe Feedback in RLHF
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencoder
  5. 2023Safer-Instruct: Aligning Language Models with Automated Preference Data
  6. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  8. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety