mlsanityData to 14 Oct 2026

Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attacks

arXiv 2410.02220 ↗First posted 3 Oct 2024cs.CR cs.AIcited by 3

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are widely adapted for downstream applications through fine-tuning, a process named customization. However, recent studies have identified a vulnerability during this process, where malicious samples can compromise the robustness of LLMs and amplify harmful behaviors-an attack commonly referred to as jailbreaking. To address this challenge, we propose an adaptive data curation approach allowing any text to be curated to enhance its effectiveness in counteracting harmful samples during customization. To avoid the need for additional defensive modules, we further introduce a comprehensive mitigation framework spanning the lifecycle of the customization process: before customization to immunize LLMs against future jailbreak attempts, during customization to neutralize risks, and after customization to restore compromised models. Experimental results demonstrate a significant reduction in jailbreaking effects, achieving up to a 100% success rate in generating safe responses. By combining adaptive data curation with lifecycle-based mitigation strategies, this work represents a solid step forward in mitigating jailbreaking risks and ensuring the secure adaptation of LLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 46 of its 67 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

46 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  2. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information20 Nov
  3. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
28 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2020Aligning AI With Shared Human Valuescited by 723
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  6. 2020Artificial Intelligence, Values and Alignmentcited by 448
  7. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  9. 2023Language Models can Solve Computer Taskscited by 372
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  11. 2021True Few-Shot Learning with Language Modelscited by 344
  12. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  13. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  14. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  15. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  16. 2024Capabilities of Gemini Models in Medicinecited by 237
  17. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  18. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  19. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  20. 2023InferFix: End-to-End Program Repair with LLMscited by 114
  21. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  22. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
  23. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 68
  24. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  25. 2024CoD, Towards an Interpretable Medical Agent using Chain of Diagnosiscited by 43
  26. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  27. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Frameworkcited by 30
  28. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  2. 2024Watermark under Fire: A Robustness Evaluation of LLM Watermarkingcited by 9
  3. 2026Understanding and Mitigating Inference-Time Overreliance Using Agentic Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  2. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
  3. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense
  4. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  5. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks
  6. 2025LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges
  7. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  8. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?