mlsanityData to 7 Oct 2026

Representation Noising: A Defence Mechanism Against Harmful Finetuning

arXiv 2405.14577 ↗First posted 23 May 2024cs.CL cs.LGcited by 93

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its ``depth'': the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 58 of its 72 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

58 of 72 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024A safety realignment framework via subspace-oriented model fusion for large language models15 May
  2. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  3. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 54

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  12. 2022Red Teaming Language Models with Language Modelscited by 983
38 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  3. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  9. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  11. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  12. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  13. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  14. 2023Large Language Model Alignment: A Surveycited by 241
  15. 2021Fast Yet Effective Machine Unlearningcited by 209
  16. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 188
  17. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  18. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  19. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  20. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  21. 2024Stealing Part of a Production Language Modelcited by 152
  22. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  23. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  24. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  25. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  26. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  27. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  28. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  29. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
  30. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  31. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  32. 2021Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorizationcited by 50
  33. 2019Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialoguecited by 35
  34. 2024Immunization against harmful fine-tuning attackscited by 33
  35. 2019ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversationcited by 31
  36. 2023Exploiting Novel GPT-4 APIscited by 27
  37. 2023Will releasing the weights of future large language models grant widespread access to pandemic agents?cited by 25
  38. 2019Learnability for the Information Bottleneckcited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 41; 2026: 44.

93 papers.

Sort:

By area:

  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  2. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  3. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  4. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  5. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  6. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  7. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  8. 2025Representation Bending for Large Language Model Safetycited by 28
  9. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  10. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  11. 2024Locking Down the Finetuned LLMs Safetycited by 22
  12. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
38 more
  1. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  2. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  3. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  4. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
  5. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  6. 2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
  7. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
  8. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  9. 2025Distillation Robustifies Unlearningcited by 11
  10. 2026Shaping capabilities with token-level data filteringcited by 11
  11. 2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
  12. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
  13. 2025Self-Destructive Language Modelcited by 10
  14. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  15. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  16. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  17. 2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
  18. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  19. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 7
  20. 2026Understanding and Preserving Safety in Fine-Tuned LLMscited by 7
  21. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  22. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  23. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 6
  24. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  25. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
  26. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  27. 2025Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methodscited by 5
  28. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 5
  29. 2025Offline Learning and Forgetting for Reasoning with Large Language Modelscited by 4
  30. 2025DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruningcited by 4
  31. 2026Gauss-Newton Unlearning for the LLM Eracited by 4
  32. 2025Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralizationcited by 3
  33. 2026Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignmentcited by 3
  34. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  35. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  36. 2025Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoringcited by 2
  37. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  38. 2025The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Modelscited by 2

50 most cited of 93 All 93 (JSON)