mlsanityData to 14 Oct 2026

Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models

arXiv 2506.16447 ↗First posted 19 Jun 2025cs.CR cs.CLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Backdoor unalignment attacks against Large Language Models (LLMs) enable the stealthy compromise of safety alignment using a hidden trigger while evading normal safety auditing. These attacks pose significant threats to the applications of LLMs in the real-world Large Language Model as a Service (LLMaaS) setting, where the deployed model is a fully black-box system that can only interact through text. Furthermore, the sample-dependent nature of the attack target exacerbates the threat. Instead of outputting a fixed label, the backdoored LLM follows the semantics of any malicious command with the hidden trigger, significantly expanding the target space. In this paper, we introduce BEAT, a black-box defense that detects triggered samples during inference to deactivate the backdoor. It is motivated by an intriguing observation (dubbed the probe concatenate effect), where concatenated triggered samples significantly reduce the refusal rate of the backdoored LLM towards a malicious probe, while non-triggered samples have little effect. Specifically, BEAT identifies whether an input is triggered by measuring the degree of distortion in the output distribution of the probe before and after concatenation with the input. Our method addresses the challenges of sample-dependent targets from an opposite perspective. It captures the impact of the trigger on the refusal signal (which is sample-independent) instead of sample-specific successful attack behaviors. It overcomes black-box access limitations by using multiple sampling to approximate the output distribution. Extensive experiments are conducted on various backdoor attacks and LLMs (including the closed-source GPT-3.5-turbo), verifying the effectiveness and efficiency of our defense. Besides, we also preliminarily verify that BEAT can effectively defend against popular jailbreak attacks, as they can be regarded as 'natural backdoors'.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 64 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

52 of 64 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  4. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  5. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  6. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2024Mixtral of Expertscited by 1,823
  8. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  9. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  10. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  11. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  12. 2020Backdoor Learning: A Surveycited by 426
31 more
  1. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  2. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  3. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  4. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 158
  5. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  6. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  7. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  8. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 104
  9. 2023SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistencycited by 97
  10. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  11. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  12. 2019Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networkscited by 71
  13. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classificationcited by 60
  14. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  15. 2022Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defensecited by 47
  16. 2023UNICORN: A Unified Backdoor Trigger Inversion Frameworkcited by 46
  17. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 39
  18. 2024IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistencycited by 39
  19. 2024Immunization against harmful fine-tuning attackscited by 35
  20. 2021Defending Against Backdoor Attacks in Natural Language Generationcited by 33
  21. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attackscited by 33
  22. 2023Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Soundcited by 30
  23. 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 28
  24. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 28
  25. 2024CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Modelscited by 24
  26. 2025REFINE: Inversion-Free Backdoor Defense via Model Reprogrammingcited by 22
  27. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 21
  28. 2023Backdoor Attack with Sparse and Invisible Triggercited by 20
  29. 2022Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attackscited by 19
  30. 2023ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLPcited by 19
  31. 2024Exploring Backdoor Vulnerabilities of Chat Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 16
  2. 2026The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggerscited by 14
  3. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  4. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
  5. 2025Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMscited by 7
  6. 2025Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearningcited by 6
  7. 2025Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attackcited by 5
  8. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  9. 2025PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMscited by 3
  10. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMscited by 2
  11. 2025SLIP: Soft Label Mechanism and Key-Extraction-Guided CoT-based Defense Against Instruction Backdoor in APIscited by 1
  12. 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoningcited by 1
7 more
  1. 2026Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Headscited by 1
  2. 2026STEP: Detecting Audio Backdoor Attacks via Stability-based Trigger Exposure Profilingcited by 1
  3. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  4. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
  5. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0
  6. 2026Localize-and-Detect: Auditing Task-Level Poisoning in Instruction-Tuned Modelscited by 0
  7. 2026Towards Merging-conditional Backdoor Unalignment against Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Model Plants the Trigger: Answer-Side Backdoor Attacks in Multi-Turn Large Language Models
  2. 2026An Empirical Study of Output-to-Input Loops for Black-Box Backdoor Detection in Fine-Tuned Open-Weight LLMs
  3. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
  4. 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs
  5. 2024Neutralizing Backdoors through Information Conflicts for Large Language Models
  6. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models
  7. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
  8. 2026Language Triggers Hijack Language Circuits: A Mechanistic Analysis of Backdoor Behaviors in Large Language Models