mlsanityData to 14 Oct 2026

Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach

arXiv 2510.01342 ↗First posted 1 Oct 2025cs.CRcited by 2

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

With the rapid advancement of large language models (LLMs), ensuring their safe use becomes increasingly critical. Fine-tuning is a widely used method for adapting models to downstream tasks, yet it is vulnerable to jailbreak attacks. However, most existing studies focus on overly simplified attack scenarios, limiting their practical relevance to real-world defense settings. To make this risk concrete, we present a three-pronged jailbreak attack and evaluate it against provider defenses under a dataset-only black-box fine-tuning interface. In this setting, the attacker can only submit fine-tuning data to the provider, while the provider may deploy defenses across stages: (1) pre-upload data filtering, (2) training-time defensive fine-tuning, and (3) post-training safety audit. Our attack combines safety-styled prefix/suffix wrappers, benign lexical encodings (underscoring) of sensitive tokens, and a backdoor mechanism, enabling the model to learn harmful behaviors while individual datapoints appear innocuous. Extensive experiments demonstrate the effectiveness of our approach. In real-world deployment, our method successfully jailbreaks GPT-4.1 and GPT-4o on the OpenAI platform with attack success rates above 97% for both models. Our code is available at https://github.com/lxf728/tri-pronged-ft-attack.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 41 of its 50 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

41 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility15 Jul
  2. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection1 Nov
  3. 2024Tamper-Resistant Safeguards for Open-Weight LLMs1 Aug
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
25 more
  1. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  2. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  4. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  5. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  6. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  7. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  8. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  9. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  10. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  11. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  12. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  13. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  14. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 62
  15. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 44
  16. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  17. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  18. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 37
  19. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderationcited by 29
  20. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 20
  21. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
  22. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 19
  23. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  24. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
  25. 2024Enhancing AI Safety Through the Fusion of Low Rank Adapterscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 8
  2. 2026SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models
  2. 2025SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Intervention
  3. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
  4. 2024Prefix Guidance: A Steering Wheel for Large Language Models to Defend Against Jailbreak Attacks
  5. 2026SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement
  6. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
  7. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models
  8. 2026How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions