mlsanityData to 7 Oct 2026

Configurable Safety Tuning of Language Models with Synthetic Preference Data

arXiv 2404.00495 ↗First posted 30 Mar 2024cs.CL cs.AIcited by 12

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

State-of-the-art language model fine-tuning techniques, such as Direct Preference Optimization (DPO), restrict user control by hard-coding predefined behaviors into the model. To address this, we propose a novel method, Configurable Safety Tuning (CST), that augments DPO using synthetic preference data to facilitate flexible safety configuration of LLMs at inference time. CST overcomes the constraints of vanilla DPO by introducing a system prompt specifying safety configurations, enabling LLM deployers to disable/enable safety preferences based on their need, just changing the system prompt. Our experimental evaluations indicate that CST successfully manages different safety configurations and retains the original functionality of LLMs, showing it is a robust method for configurable deployment. Data and models available at https://github.com/vicgalle/configurable-safety-tuning

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 5 of its 5 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

5 of 5 references.

References · 5

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 13
  5. 2023Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspectivecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 1.

12 papers.

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  3. 2025Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inferencecited by 18
  4. 2025SoK: Large Language Model Copyright Auditing via Fingerprintingcited by 14
  5. 2024You Know What I'm Saying: Jailbreak Attack via Implicit Referencecited by 13
  6. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 7
  7. 2025MetaSC: Test-Time Safety Specification Optimization for Language Modelscited by 4
  8. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  9. 2025Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settingscited by 3
  10. 2025Prompting for Performance: Exploring LLMs for Configuring Softwarecited by 1
  11. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systemscited by 1
  12. 2025SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization
  2. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
  4. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
  5. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  6. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  7. 2026Improving Safety Alignment via Balanced Direct Preference Optimization
  8. 2026Understanding and Preserving Safety in Fine-Tuned LLMs