# Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes

arXiv: https://arxiv.org/abs/2404.04392 · HTML: https://mlsanity.com/paper/2404.04392/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 18 of its 18 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 18 of 18)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [ProPILE: Probing Privacy Leakage in Large Language Models](https://mlsanity.com/paper/2307.01881/) (2023)
- [AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2310.15140/) (2023)
- [Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models](https://mlsanity.com/paper/2308.10462/) (2023)
- [MALCOM: Generating Malicious Comments to Attack Neural Fake News Detection Models](https://mlsanity.com/paper/2009.01048/) (2020)
- [PETGEN: Personalized Text Generation Attack on Deep Sequence Embedding-based Classification Models](https://mlsanity.com/paper/2109.06777/) (2021)

## Built on by (most cited first, 14 of 14)

- [On the Impact of Fine-Tuning on Chain-of-Thought Reasoning](https://mlsanity.com/paper/2411.15382/) (2024)
- [JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models](https://mlsanity.com/paper/2406.09321/) (2024)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMs](https://mlsanity.com/paper/2410.12864/) (2024)
- [HarmLevelBench: Evaluating Harm-Level Compliance and the Impact of Quantization on Model Alignment](https://mlsanity.com/paper/2411.06835/) (2024)
- [Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards](https://mlsanity.com/paper/2505.16789/) (2025)
- [ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling](https://mlsanity.com/paper/2601.21198/) (2026)
- [Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models](https://mlsanity.com/paper/2506.20251/) (2025)
- [EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering](https://mlsanity.com/paper/2511.19523/) (2025)
- [Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection](https://mlsanity.com/paper/2601.12033/) (2026)
- [Lattice: Generative Guardrails for Conversational Agents](https://mlsanity.com/paper/2601.17481/) (2026)
- [SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Models](https://mlsanity.com/paper/2512.14481/) (2025)
- [Gradient-Controlled Decoding: A Safety Guardrail for LLMs with Dual-Anchor Steering](https://mlsanity.com/paper/2604.05179/) (2026)
- [On the Efficiency-Safety Dilemma in Large Reasoning Models](https://mlsanity.com/paper/2609.23587/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have gained widespread adoption across various domains, including chatbots and auto-task completion agents. However, these models are susceptible to safety vulnerabilities such as jailbreaking, prompt injection, and privacy leakage attacks. These vulnerabilities can lead to the generation of malicious content, unauthorized actions, or the disclosure of confidential information. While foundational LLMs undergo alignment training and incorporate safety measures, they are often subject to fine-tuning, or doing quantization resource-constrained environments. This study investigates the impact of these modifications on LLM safety, a critical consideration for building reliable and secure AI systems. We evaluate foundational models including Mistral, Llama series, Qwen, and MosaicML, along with their fine-tuned variants. Our comprehensive analysis reveals that fine-tuning generally increases the success rates of jailbreak attacks, while quantization has variable effects on attack success rates. Importantly, we find that properly implemented guardrails significantly enhance resistance to jailbreak attempts. These findings contribute to our understanding of LLM vulnerabilities and provide insights for developing more robust safety strategies in the deployment of language models.