mlsanityData to 14 Oct 2026

Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack

arXiv 2312.06924 ↗First posted 12 Dec 2023cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent developments in balancing the usefulness and safety of Large Language Models (LLMs) have raised a critical question: Are mainstream NLP tasks adequately aligned with safety consideration? Our study, focusing on safety-sensitive documents obtained through adversarial attacks, reveals significant disparities in the safety alignment of various NLP tasks. For instance, LLMs can effectively summarize malicious long documents but often refuse to translate them. This discrepancy highlights a previously unidentified vulnerability: attacks exploiting tasks with weaker safety alignment, like summarization, can potentially compromise the integrity of tasks traditionally deemed more robust, such as translation and question-answering (QA). Moreover, the concurrent use of multiple NLP tasks with lesser safety alignment increases the risk of LLMs inadvertently processing harmful content. We demonstrate these vulnerabilities in various safety-aligned LLMs, particularly Llama2 models, Gemini and GPT-4, indicating an urgent need for strengthening safety alignments across a broad spectrum of NLP tasks.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 45 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

24 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  7. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  10. 2023Poisoning Language Models During Instruction Tuningcited by 269
  11. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  12. 2023On the Exploitability of Instruction Tuningcited by 116
5 more
  1. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  2. 2022Generative Language Models for Paragraph-Level Question Generationcited by 28
  3. 2021SHARE: a System for Hierarchical Assistive Recipe Editingcited by 15
  4. 2023White-Box Multi-Objective Adversarial Attack on Dialogue Generationcited by 14
  5. 2023RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 2.

11 papers.

  1. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  2. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  3. 2024On the Vulnerability of LLM/VLM-Controlled Roboticscited by 15
  4. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14
  5. 2024Cross-Task Defense: Instruction-Tuning LLMs for Content Safetycited by 8
  6. 2025Automatic LLM Red Teamingcited by 5
  7. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  8. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 3
  9. 2025MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignmentcited by 2
  10. 2026PIMbot: A Self-Adaptive Attack Framework for Adversarial Manipulation of Multi-Robot Reinforcement Learningcited by 1
  11. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  2. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals
  3. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
  4. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation
  5. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  6. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  7. 2025Advancing LLM Safe Alignment with Safety Representation Ranking
  8. 2023Fake Alignment: Are LLMs Really Aligned Well?