mlsanityData to 14 Oct 2026

Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment

arXiv 2503.18991 ↗First posted 23 Mar 2025cs.CL cs.AI cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Alignment is vital for safely deploying large language models (LLMs). Existing techniques are either reward-based (training a reward model on preference pairs and optimizing with reinforcement learning) or reward-free (directly fine-tuning on ranked outputs). Recent research shows that well-tuned reward-based pipelines remain the most robust, and single-response demonstrations can outperform pairwise preference data. However, there still exist two key challenges: (1) imbalanced safety datasets that overrepresent common hazards while neglecting long-tail threats; and (2) static reward models that ignore task difficulty, limiting optimization efficiency and attainable gains. To address these limitations, we propose DR-IRL, which Dynamically adjusts Rewards through Inverse Reinforcement Learning. We first train category-specific reward models using a balanced safety dataset of seven harmful categories as demonstration via IRL. Then we enhance Group Relative Policy Optimization (GRPO) by introducing dynamic reward scaling: adjusting rewards by task difficulty, data-level hardness by text encoder cosine similarity, and model-level responsiveness by reward gaps. Extensive experiments across various benchmarks and LLMs demonstrate that DR-IRL outperforms all baseline methods in safety alignment while maintaining usefulness.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 65 of its 82 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

65 of 82 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences15 Mar
  2. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment28 May
  3. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs15 Apr
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  7. 2022Models of human preference for learning reward functions5 Jun
  8. 2018Reward learning from human preferences and demonstrations in Atari15 Nov

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
38 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  6. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  11. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  12. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  13. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  14. 2024A StrongREJECT for Empty Jailbreakscited by 385
  15. 2024Measuring short-form factuality in large language modelscited by 373
  16. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  17. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 297
  18. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  19. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  20. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  21. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  22. 2023Large Language Model Alignment: A Surveycited by 243
  23. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  24. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  25. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  26. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  27. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  28. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  29. 2023Mitigating the Alignment Tax of RLHFcited by 183
  30. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
  31. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  32. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  33. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  34. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 54
  35. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
  36. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialoguecited by 33
  37. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  38. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 8.

20 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  2. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  3. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  4. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  5. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
  6. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  7. 2025Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMscited by 9
  8. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  9. 2025Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMscited by 2
  10. 2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
  11. 2026Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigationcited by 1
  12. 2025DMRL: Data- and Model-aware Reward Learning for Data Extractioncited by 0
8 more
  1. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Searchcited by 0
  2. 2025Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection: A VAE-Enhanced Reinforcement Learning Approachcited by 0
  3. 2025Information-Dense Reasoning for Efficient and Auditable Security Alert Triagecited by 0
  4. 2026ShellForge: Adversarial Co-Evolution of Webshell Generation and Multi-View Detection for Robust Webshell Defensecited by 0
  5. 2026Membership Inference for Contrastive Pre-training Models with Text-only PII Queriescited by 0
  6. 2026MESA: Improving MoE Safety Alignment via Decentralized Expertisecited by 0
  7. 2026Coherence Maximization Improves Pluralistic Alignmentcited by 0
  8. 2026Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
  2. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  3. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  4. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
  5. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  6. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  7. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL
  8. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs