mlsanityData to 7 Oct 2026

Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors

arXiv 2604.12359 ↗First posted 14 Apr 2026cs.CR cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines, yet this deployment also enlarges the supply-chain attack surface: adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present. Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors by directly modifying model weights to map a trigger to an attacker-specified response. However, existing methods typically optimize a token-level mapping that forces an affirmative prefix (e.g., ``Sure''), which does not guarantee sustained harmful output -- the model may begin with apparent agreement yet revert to safety-aligned refusal within a few decoding steps. We address this reliability gap by shifting the backdoor objective from surface tokens to internal representations. We extract a steering vector that captures the difference between compliant and refusal behaviors, and compile it into a persistent weight modification that activates only when the trigger is present. To preserve stealthiness and benign utility, we impose a null-space constraint so that the injected edit remains dormant on clean inputs. The method is efficient, requiring only a small set of examples and admitting a closed-form solution. Across multiple safety-aligned LLMs and jailbreak benchmarks, our method achieves high triggered attack success while maintaining non-triggered safety and general utility.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 52 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

23 of 52 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
11 more
  1. 2023Poisoning Language Models During Instruction Tuningcited by 254
  2. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  3. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  4. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  5. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
  6. 2024BadEdit: Backdooring large language models by model editingcited by 84
  7. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  8. 2024Post-Training Attribute Unlearning in Recommender Systemscited by 23
  9. 2025Injecting Universal Jailbreak Backdoors into LLMs in Minutescited by 13
  10. 2025Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?cited by 8
  11. 2026FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attackscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
  2. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models
  3. 2026Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs
  4. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
  5. 2026The Model Plants the Trigger: Answer-Side Backdoor Attacks in Multi-Turn Large Language Models
  6. 2026SpecGuard: Inference-Time Backdoor Detection For Free
  7. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
  8. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defense