mlsanityData to 14 Oct 2026

The effect of fine-tuning on language model toxicity

arXiv 2410.15821 ↗First posted 21 Oct 2024cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Fine-tuning language models has become increasingly popular following the proliferation of open models and improvements in cost-effective parameter efficient fine-tuning. However, fine-tuning can influence model properties such as safety. We assess how fine-tuning can impact different open models' propensity to output toxic content. We assess the impacts of fine-tuning Gemma, Llama, and Phi models on toxicity through three experiments. We compare how toxicity is reduced by model developers during instruction-tuning. We show that small amounts of parameter-efficient fine-tuning on developer-tuned models via low-rank adaptation on a non-adversarial dataset can significantly alter these results across models. Finally, we highlight the impact of this in the wild, demonstrating how toxicity rates of models fine-tuned by community contributors can deviate in hard-to-predict ways.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

12 of 21 references.

References · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  5. 2023Fine-tuning Language Models for Factualitycited by 221
  6. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  7. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  8. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  9. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  10. 2024LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystemcited by 24
  11. 2023Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?cited by 10
  12. 2024Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  2. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
  3. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMscited by 10
  4. 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigationcited by 9
  5. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
  6. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  7. 2025ConSens: Assessing context grounding in open-book question answeringcited by 4
  8. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  9. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 3
  10. 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorderscited by 1
  11. 2026TrojanPraise: Jailbreak LLMs via Benign Fine-Tuningcited by 1
  12. 2026Can LLM Safety Be Ensured by Constraining Parameter Regions?cited by 0
4 more
  1. 2026Few-Shot Truly Benign DPO Attack for Jailbreaking LLMscited by 0
  2. 2026A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Modelscited by 0
  3. 2026The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuningcited by 0
  4. 2026Defending Against Harmful Supervision Hidden in Benign Samplescited by 0