# The Hidden Costs of Domain Fine-Tuning: Pii-Bearing Data Degrades Safety and Increases Leakage

arXiv: https://arxiv.org/abs/2603.00061 · HTML: https://mlsanity.com/paper/2603.00061/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates](https://mlsanity.com/paper/2402.18540/) (2024)
- [Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack](https://mlsanity.com/paper/2405.18641/) (2024)
- [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [Secure LLM Fine-Tuning via Safety-Aware Probing](https://mlsanity.com/paper/2505.16737/) (2025)
- [Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs](https://mlsanity.com/paper/2507.03662/) (2025)
- [Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models](https://mlsanity.com/paper/2508.14062/) (2025)

## Built on by (most cited first, 1 of 1)

- [Mitigating Private Data Leakage in LLMs with Whiteout](https://mlsanity.com/paper/2610.02418/) (2026)

## Abstract (arXiv)

Domain fine-tuning is a common path to deploy small instruction-tuned language models as customer-support assistants, yet its effects on safety-aligned behavior and privacy are not well understood. In real deployments, such assistants receive a mixture of benign in-domain requests and out-of-domain user queries that are emotional, philosophical, or adversarial. Even when the target domain is benign, specialization may shift model behavior in ways that weaken refusal, increase harmful compliance, and induce privacy leakage. We present a controlled empirical study of how training data composition (presence vs. removal of PII) and fine-tuning configuration (role-swapping (RS)) shape safety and out-of-domain behavior in open-source chat models up to 8B parameters. We fine-tune each model on 5,000 real booking-support message pairs under three settings: textscNoPII-NoRS, textscPII-NoRS, and textscPII-RS (role-swapped). We evaluate safety using textscSORRY-Bench~citexie2024sorry adversarial prompts and assess out-of-domain behavior using a suite of philosophical questions~citebetley2025emergent. Across models, domain fine-tuning causes a large distributional shift from high-quality refusals toward harmful compliance on textscSORRY-Bench, with the most severe degradation when PII is present in the fine-tuning data. For example, macro-averaged strong refusal drops from 42.6% in base models to single digits after fine-tuning, while PII-bearing runs additionally exhibit double-digit rates of harmful responses with PII leakage. On philosophical queries, fine-tuned models frequently exhibit domain anchoring and, when trained with PII, leak sensitive identifiers in irrelevant contexts. Role-swapping partially mitigates PII leakage but does not reliably restore refusal behavior.