mlsanityData to 7 Oct 2026

Jailbreaking as a Reward Misspecification Problem

arXiv 2406.14393 ↗First posted 20 Jun 2024cs.LG cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The widespread adoption of large language models (LLMs) has raised concerns about their safety and reliability, particularly regarding their vulnerability to adversarial attacks. In this paper, we propose a novel perspective that attributes this vulnerability to reward misspecification during the alignment process. This misspecification occurs when the reward function fails to accurately capture the intended behavior, leading to misaligned model outputs. We introduce a metric ReGap to quantify the extent of reward misspecification and demonstrate its effectiveness and robustness in detecting harmful backdoor prompts. Building upon these insights, we present ReMiss, a system for automated red teaming that generates adversarial prompts in a reward-misspecified space. ReMiss achieves state-of-the-art attack success rates on the AdvBench benchmark against various target aligned LLMs while preserving the human readability of the generated prompts. Furthermore, these attacks on open-source models demonstrate high transferability to closed-source models like GPT-4o and out-of-distribution tasks from HarmBench. Detailed analysis highlights the unique advantages of the proposed reward misspecification objective compared to previous methods, offering new insights for improving LLM safety and robustness.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 59 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

49 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusal25 Apr
  2. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
32 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  13. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  14. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  15. 2024A StrongREJECT for Empty Jailbreakscited by 352
  16. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  17. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  18. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  19. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  20. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  21. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  22. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  23. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  24. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  25. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  26. 2024The Ethics of Advanced AI Assistantscited by 103
  27. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  28. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  29. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  30. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  31. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27
  32. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shiftcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 3.

11 papers.

  1. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
  2. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  3. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  4. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 18
  5. 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
  6. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  7. 2025Jailbreaking with Universal Multi-Promptscited by 2
  8. 2026Mitigating loss of control in advanced AI systems through instrumental goal trajectoriescited by 1
  9. 2025An Aristotelian ontology of instrumental goals: Structural features to be managed and not failures to be eliminatedcited by 0
  10. 2026New Wide-Net-Casting Jailbreak Attacks Risk Large Modelscited by 0
  11. 2026In-Context Learning as Implicit Policy Gradientcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Stealing Attack on Large Language Models
  2. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  3. 2025Adversarial Training of Reward Models
  4. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  5. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  6. 2026Understanding the Effects of Safety Unalignment on Large Language Models
  7. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
  8. 2024Investigating Adversarial Trigger Transfer in Large Language Models