mlsanityData to 7 Oct 2026

HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment

arXiv 2510.15499 ↗First posted 17 Oct 2025cs.CRcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have gained significant attention due to their objective and verifiable reward signals, demonstrating strong performance in reasoning and code generation tasks. However, the potential safety risks associated with RLVR remain underexplored. This paper presents HarmRLVR, the first systematic investigation into the alignment reversibility risk of RLVR. We show that safety alignment can be rapidly reversed using GRPO with merely 64 harmful prompts without responses, causing models to readily comply with harmful instructions. Across five models from Llama, Qwen, and DeepSeek, we empirically demonstrate that RLVR-based attacks elevate the average harmfulness score to 4.94 with an attack success rate of 96.01%, significantly outperforming harmful fine-tuning while preserving general capabilities. Our findings reveal that RLVR can be efficiently exploited for harmful alignment, posing serious threats to open-source model safety. Please see our code at https://github.com/lyxx2535/HarmRLVR.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 41 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

29 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment24 Jul
  2. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  3. 2024Evaluating Defences against Unsafe Feedback in RLHF19 Sep
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  6. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
9 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  4. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  5. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  6. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  7. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
  8. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  9. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  2. 2026Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreakscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs
  2. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward
  3. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment
  4. 2026Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
  5. 2025LLM Misalignment via Adversarial RLHF Platforms
  6. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
  7. 2026Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
  8. 2026Alignment Risks from Capability-Seeking RL Training