# Configurable Safety Tuning of Language Models with Synthetic Preference Data

arXiv: https://arxiv.org/abs/2404.00495 · HTML: https://mlsanity.com/paper/2404.00495/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 5 of its 5 references that are in the map.
- First posted on arXiv 30 Mar 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)

## Builds on (most cited first, 5 of 5)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Suppressing Pink Elephants with Direct Principle Feedback](https://mlsanity.com/paper/2402.07896/) (2024)
- [Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspective](https://mlsanity.com/paper/2312.01957/) (2023)

## Built on by (most cited first, 14 of 14)

- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference](https://mlsanity.com/paper/2508.08438/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [You Know What I'm Saying: Jailbreak Attack via Implicit Reference](https://mlsanity.com/paper/2410.03857/) (2024)
- [You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense](https://mlsanity.com/paper/2501.12210/) (2025)
- [Configurable Preference Tuning with Rubric-Guided Synthetic Data](https://mlsanity.com/paper/2506.11702/) (2025)
- [MetaSC: Test-Time Safety Specification Optimization for Language Models](https://mlsanity.com/paper/2502.07985/) (2025)
- [Efficient Safety Retrofitting Against Jailbreaking for LLMs](https://mlsanity.com/paper/2502.13603/) (2025)
- [Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings](https://mlsanity.com/paper/2505.15353/) (2025)
- [Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense](https://mlsanity.com/paper/2601.03594/) (2026)
- [Prompting for Performance: Exploring LLMs for Configuring Software](https://mlsanity.com/paper/2507.09790/) (2025)
- [PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems](https://mlsanity.com/paper/2603.10726/) (2026)
- [SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models](https://mlsanity.com/paper/2509.26345/) (2025)

## Abstract (arXiv)

State-of-the-art language model fine-tuning techniques, such as Direct Preference Optimization (DPO), restrict user control by hard-coding predefined behaviors into the model. To address this, we propose a novel method, Configurable Safety Tuning (CST), that augments DPO using synthetic preference data to facilitate flexible safety configuration of LLMs at inference time. CST overcomes the constraints of vanilla DPO by introducing a system prompt specifying safety configurations, enabling LLM deployers to disable/enable safety preferences based on their need, just changing the system prompt. Our experimental evaluations indicate that CST successfully manages different safety configurations and retains the original functionality of LLMs, showing it is a robust method for configurable deployment. Data and models available at https://github.com/vicgalle/configurable-safety-tuning