mlsanityData to 14 Oct 2026

Rethinking Deep Alignment Through The Lens Of Incomplete Learning

arXiv 2511.12155 ↗First posted 15 Nov 2025cs.LGcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large language models exhibit systematic vulnerabilities to adversarial attacks despite extensive safety alignment. We provide a mechanistic analysis revealing that position-dependent gradient weakening during autoregressive training creates signal decay, leading to incomplete safety learning where safety training fails to transform model preferences in later response regions fully. We introduce base-favored tokens -- vocabulary elements where base models assign higher probability than aligned models -- as computational indicators of incomplete safety learning and develop a targeted completion method that addresses undertrained regions through adaptive penalties and hybrid teacher distillation. Experimental evaluation across Llama and Qwen model families demonstrates dramatic improvements in adversarial robustness, with 48--98% reductions in attack success rates while preserving general capabilities. These results establish both a mechanistic understanding and practical solutions for fundamental limitations in safety alignment methodologies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 35 of its 49 references that are in the map. First posted on arXiv 15 Nov 2025.

Builds on

35 of 49 references.

References · 35

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
23 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  10. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  13. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  14. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  15. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  16. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  17. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  18. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  19. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  20. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  21. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 36
  22. 2025Extracting and Understanding the Superficial Knowledge in Alignmentcited by 5
  23. 2024Risk-Sensitive Online Algorithmscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals
  2. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbations
  3. 2026Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
  4. 2025Safety Alignment Should Be Made More Than Just A Few Attention Heads
  5. 2026Target-free Latent Safety Alignment
  6. 2026When Autoregressive Consistency Hurts Safety Alignment
  7. 2026Understanding the Effects of Safety Unalignment on Large Language Models
  8. 2025Improving LLM Safety Alignment with Dual-Objective Optimization