# Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack

arXiv: https://arxiv.org/abs/2312.06924 · HTML: https://mlsanity.com/paper/2312.06924/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023)
- [On the Exploitability of Instruction Tuning](https://mlsanity.com/paper/2306.17194/) (2023)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [SHARE: a System for Hierarchical Assistive Recipe Editing](https://mlsanity.com/paper/2105.08185/) (2021)
- [White-Box Multi-Objective Adversarial Attack on Dialogue Generation](https://mlsanity.com/paper/2305.03655/) (2023)
- [RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models](https://mlsanity.com/paper/2309.06619/) (2023)

## Built on by (most cited first, 11 of 11)

- [Weak-to-Strong Jailbreaking on Large Language Models](https://mlsanity.com/paper/2401.17256/) (2024)
- [The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness](https://mlsanity.com/paper/2401.00287/) (2023)
- [On the Vulnerability of LLM/VLM-Controlled Robotics](https://mlsanity.com/paper/2402.10340/) (2024)
- [Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation](https://mlsanity.com/paper/2503.08195/) (2025)
- [Cross-Task Defense: Instruction-Tuning LLMs for Content Safety](https://mlsanity.com/paper/2405.15202/) (2024)
- [Automatic LLM Red Teaming](https://mlsanity.com/paper/2508.04451/) (2025)
- [SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses](https://mlsanity.com/paper/2510.15476/) (2025)
- [LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems](https://mlsanity.com/paper/2507.21276/) (2025)
- [MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment](https://mlsanity.com/paper/2510.03283/) (2025)
- [PIMbot: A Self-Adaptive Attack Framework for Adversarial Manipulation of Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2605.23027/) (2026)
- [Robust Critics: Defending LLMs Against Multi-Turn Attacks](https://mlsanity.com/paper/2607.20472/) (2026)

## Abstract (arXiv)

Recent developments in balancing the usefulness and safety of Large Language Models (LLMs) have raised a critical question: Are mainstream NLP tasks adequately aligned with safety consideration? Our study, focusing on safety-sensitive documents obtained through adversarial attacks, reveals significant disparities in the safety alignment of various NLP tasks. For instance, LLMs can effectively summarize malicious long documents but often refuse to translate them. This discrepancy highlights a previously unidentified vulnerability: attacks exploiting tasks with weaker safety alignment, like summarization, can potentially compromise the integrity of tasks traditionally deemed more robust, such as translation and question-answering (QA). Moreover, the concurrent use of multiple NLP tasks with lesser safety alignment increases the risk of LLMs inadvertently processing harmful content. We demonstrate these vulnerabilities in various safety-aligned LLMs, particularly Llama2 models, Gemini and GPT-4, indicating an urgent need for strengthening safety alignments across a broad spectrum of NLP tasks.