mlsanityData to 7 Oct 2026

Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs

arXiv 2412.06843 ↗First posted 7 Dec 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

36 of 43 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Mistral 7Bcited by 3,065
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
24 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2024Mixtral of Expertscited by 1,785
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  10. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  13. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  14. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  15. 2022Predictability and Surprise in Large Generative Modelscited by 209
  16. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  17. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  18. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  19. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  20. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  21. 2024Aligning Crowd Feedback via Distributional Preference Reward Modelingcited by 26
  22. 2023EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modelingcited by 7
  23. 2023Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learningcited by 7
  24. 2024Unlocking Large Language Model's Planning Capabilities with Maximum Diversity Fine-tuningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  2. 2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinementcited by 3
  3. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  4. 2025How to Trick Your AI TA: A Systematic Study of Academic Jailbreaking in LLM Code Evaluationcited by 0