Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
Abstract (arXiv)
Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines, yet this deployment also enlarges the supply-chain attack surface: adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present. Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors by directly modifying model weights to map a trigger to an attacker-specified response. However, existing methods typically optimize a token-level mapping that forces an affirmative prefix (e.g., ``Sure''), which does not guarantee sustained harmful output -- the model may begin with apparent agreement yet revert to safety-aligned refusal within a few decoding steps. We address this reliability gap by shifting the backdoor objective from surface tokens to internal representations. We extract a steering vector that captures the difference between compliant and refusal behaviors, and compile it into a persistent weight modification that activates only when the trigger is present. To preserve stealthiness and benign utility, we impose a null-space constraint so that the injected edit remains dormant on clean inputs. The method is efficient, requiring only a small set of examples and admitting a closed-form solution. Across multiple safety-aligned LLMs and jailbreak benchmarks, our method achieves high triggered attack success while maintaining non-triggered safety and general utility.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 52 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
23 of 52 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2022Mass-Editing Memory in a Transformercited by 976
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
11 more
- 2023Poisoning Language Models During Instruction Tuningcited by 254
- 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
- 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
- 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
- 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
- 2024BadEdit: Backdooring large language models by model editingcited by 84
- 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
- 2024Post-Training Attribute Unlearning in Recommender Systemscited by 23
- 2025Injecting Universal Jailbreak Backdoors into LLMs in Minutescited by 13
- 2025Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?cited by 8
- 2026FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attackscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
- 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models
- 2026Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs
- 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
- 2026The Model Plants the Trigger: Answer-Side Backdoor Attacks in Multi-Turn Large Language Models
- 2026SpecGuard: Inference-Time Backdoor Detection For Free
- 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
- 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defense