mlsanityData to 7 Oct 2026

Removing RLHF Protections in GPT-4 via Fine-Tuning

arXiv 2311.05553 ↗First posted 9 Nov 2023cs.CL cs.AIcited by 165

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

As large language models (LLMs) have increased in their capabilities, so does their potential for dual use. To reduce harmful outputs, produces and vendors of LLMs have used reinforcement learning with human feedback (RLHF). In tandem, LLM vendors have been increasingly enabling fine-tuning of their most powerful models. However, concurrent work has shown that fine-tuning can remove RLHF protections. We may expect that the most powerful models currently available (GPT-4) are less susceptible to fine-tuning attacks. In this work, we show the contrary: fine-tuning allows attackers to remove RLHF protections with as few as 340 examples and a 95% success rate. These training examples can be automatically generated with weaker models. We further show that removing RLHF protections does not decrease usefulness on non-censored outputs, providing evidence that our fine-tuning strategy does not decrease usefulness despite using weaker models to generate training data. Our results show the need for further research on protections on LLMs.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 19 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

12 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  7. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  8. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  9. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 68; 2025: 54; 2026: 41.

165 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  4. 2024A StrongREJECT for Empty Jailbreakscited by 352
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  7. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  8. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  9. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  10. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  11. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  12. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
38 more
  1. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  2. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  3. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2024LLM Agents can Autonomously Hack Websitescited by 108
  6. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  7. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  8. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  9. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  10. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  11. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  12. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  13. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  14. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  15. 2024Safeguarding Large Language Models: A Surveycited by 73
  16. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  17. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  18. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  19. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  20. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  21. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  22. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 45
  23. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  24. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  25. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  26. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  27. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  28. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  29. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
  30. 2024Immunization against harmful fine-tuning attackscited by 33
  31. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  32. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  33. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  34. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  35. 2023Exploiting Novel GPT-4 APIscited by 27
  36. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  37. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
  38. 2024Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Modelscited by 24

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity
  3. 2025LLM Misalignment via Adversarial RLHF Platforms
  4. 2025Challenges in Ensuring AI Safety in DeepSeek-R1 Models: The Shortcomings of Reinforcement Learning Strategies
  5. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  6. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  7. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
  8. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond