mlsanityData to 7 Oct 2026

Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing

arXiv 2606.00686 ↗First posted 30 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → Expert routing strategies

also in 1 more
Abstract (arXiv)

The prevailing paradigm in large language model (LLM) alignment operates via erasure, filtering unsafe data or training models to strictly refuse harmful prompts. While effective at reducing immediate toxicity, this approach fundamentally constricts the model's epistemological scope, resulting in over-cautious systems that output uninformative blanket refusals to sensitive yet benign queries. In this work, we challenge the orthodoxy that unsafe data must be discarded. We propose a dialectical approach to alignment, positing that unsafe data encodes rich, domain specific knowledge critical for nuanced, safe, and informative generation. To operationalize this, we introduce SafeMoE, a Mixture-of-Experts (MoE) framework that isolates unsafe knowledge into domain-specific Low-Rank Adapters (LoRA experts) trained exclusively on harmful corpora. To synthesize safety from these unsafe primitives, we train a lightweight gating network using a minimal, highly curated set of safe-informative responses. During inference, this router dynamically orchestrates the unsafe experts, effectively steering the generation trajectory to harness their deep domain knowledge while strictly enforcing safety constraints. Extensive empirical evaluations across stringent safety benchmarks demonstrate that SafeMoE is not only safer, achieving over a 20% relative improvement in safe response rate (more than a 15% absolute gain), but also produces more informative responses when safety and harmfulness are of paramount concern. Furthermore, the routing mechanism exhibits strong zero-shot generalization to unseen domains and broader safety tasks without domain-specific supervision. Our findings suggest a paradigm shift in alignment: true safety requires not the masking of unsafe knowledge, but its controlled integration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 55 of its 73 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

55 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 54

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2022Editing Models with Task Arithmeticcited by 1,076
38 more
  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  10. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  13. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  14. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  15. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  16. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  17. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  18. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  19. 2024Mixture of LoRA Expertscited by 179
  20. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  21. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  22. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  23. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  24. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  25. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  26. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  27. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  28. 2024Open Problems in Technical AI Governancecited by 45
  29. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  30. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  31. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  32. 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33
  33. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  34. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  35. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 24
  36. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  37. 2024The 2024 Foundation Model Transparency Indexcited by 18
  38. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  2. 2026MOSAIC: Composable Safety Alignment with Modular Control Tokens
  3. 2026Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignment
  4. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety
  5. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
  6. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  7. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  8. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspective