mlsanityData to 7 Oct 2026

Configurable Reward Model for Balanced Safety Alignment

arXiv 2605.30487 ↗First posted 28 May 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Aligning large language models (LLMs) to heterogeneous and rapidly evolving safety requirements remains a critical challenge. Existing instruction-tuned LLMs and standalone safety classifiers often fail to generalize to new safety configurations, motivating the need for Reward Models (RMs) that are explicitly configurable to changing specifications. We introduce the Configurable Safety Reward Model (CSRM), which is jointly optimized for calibrated safety compliance and reward modeling. Our approach is supported by configuration-targeted data augmentation that enforces instruction adherence while preserving relative severity structure. The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations. CSRM achieves state-of-the-art performance on recent configurable safety benchmarks, including CoSApien (94.6% F1) and DynaBench (75.8% F1), without requiring additional human annotation. When used for downstream safety alignment, CSRM yields LLMs with a significantly improved helpfulness-safety tradeoff compared to existing baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 52 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

40 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023Steering Language Models With Activation Engineeringcited by 824
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  11. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  12. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
25 more
  1. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  2. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  3. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  4. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  5. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  6. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
  7. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  8. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  9. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
  10. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  11. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  12. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  13. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  14. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Modelscited by 19
  15. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 19
  16. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answeringcited by 17
  17. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 16
  18. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedbackcited by 15
  19. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 14
  20. 2024Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibrationcited by 13
  21. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scorescited by 7
  22. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 7
  23. 2024RNR: Teaching Large Language Models to Follow Roles and Rulescited by 6
  24. 2025MetaSC: Test-Time Safety Specification Optimization for Language Modelscited by 4
  25. 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  2. 2024Superficial Safety Alignment Hypothesis
  3. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
  4. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements
  5. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  6. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
  7. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
  8. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization