mlsanityData to 14 Oct 2026

From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment

arXiv 2506.10020 ↗First posted 7 Jun 2025cs.CR cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safely aligning large language models (LLMs) often demands extensive human-labeled preference data, a process that's both costly and time-consuming. While synthetic data offers a promising alternative, current methods frequently rely on complex iterative prompting or auxiliary models. To address this, we introduce Refusal-Aware Adaptive Injection (RAAI), a straightforward, training-free, and model-agnostic framework that repurposes LLM attack techniques. RAAI works by detecting internal refusal signals and adaptively injecting predefined phrases to elicit harmful, yet fluent, completions. Our experiments show RAAI effectively jailbreaks LLMs, increasing the harmful response rate from a baseline of 2.15% to up to 61.04% on average across four benchmarks. Crucially, fine-tuning LLMs with the synthetic data generated by RAAI improves model robustness against harmful prompts while preserving general capabilities on standard tasks like MMLU and ARC. This work highlights how LLM attack methodologies can be reframed as practical tools for scalable and controllable safety alignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 52 references that are in the map. First posted on arXiv 7 Jun 2025.

Builds on

38 of 52 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable1 Mar
  2. 2024Rule Based Rewards for Language Model Safety2 Nov
  3. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements11 Oct
  4. 2024Course-Correction: Safety Alignment Using Synthetic Preferences23 Jul
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  6. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!19 Feb
  7. 2023Safer-Instruct: Aligning Language Models with Automated Preference Data15 Nov
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  10. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  8. 2023LIMA: Less Is More for Alignmentcited by 1,100
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2022Red Teaming Language Models with Language Modelscited by 1,026
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
15 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  2. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  6. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  7. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  8. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  9. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  10. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  11. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  12. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 36
  13. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 23
  14. 2024SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teamingcited by 10
  15. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedbackcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving LLM Safety Alignment with Dual-Objective Optimization
  2. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attacks
  3. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  4. 2026Fail-Closed Alignment for Large Language Models
  5. 2026Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment
  6. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
  7. 2026RAS: Measuring LLM Safety Through Refusal Alignment
  8. 2024Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment