mlsanityData to 13 Oct 2026

Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases

arXiv 2310.14303 ↗First posted 22 Oct 2023cs.CLcited by 23

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Red-teaming has been a widely adopted way to evaluate the harmfulness of Large Language Models (LLMs). It aims to jailbreak a model's safety behavior to make it act as a helpful agent disregarding the harmfulness of the query. Existing methods are primarily based on input text-based red-teaming such as adversarial prompts, low-resource prompts, or contextualized prompts to condition the model in a way to bypass its safe behavior. Bypassing the guardrails uncovers hidden harmful information and biases in the model that are left untreated or newly introduced by its safety training. However, prompt-based attacks fail to provide such a diagnosis owing to their low attack success rate, and applicability to specific models. In this paper, we present a new perspective on LLM safety research i.e., parametric red-teaming through Unalignment. It simply (instruction) tunes the model parameters to break model guardrails that are not deeply rooted in the model's behavior. Unalignment using as few as 100 examples can significantly bypass commonly referred to as CHATGPT, to the point where it responds with an 88% success rate to harmful queries on two safety benchmark datasets. On open-source models such as VICUNA-7B and LLAMA-2-CHAT 7B AND 13B, it shows an attack success rate of more than 91%. On bias evaluations, Unalignment exposes inherent biases in safety-aligned models such as CHATGPT and LLAMA- 2-CHAT where the model's responses are strongly biased and opinionated 64% of the time.

Key facts. Cited by 23 papers in the map (data through 13 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 30 of its 35 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

30 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  2. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  12. 2020Learning to summarize from human feedbackcited by 2,575
15 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,499
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 697
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 622
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 493
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 443
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 422
  8. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 335
  9. 2023Low-Resource Languages Jailbreak GPT-4cited by 306
  10. 2023The False Promise of Imitating Proprietary LLMscited by 258
  11. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 235
  12. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 193
  13. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  14. 2021Alignment of Language Agentscited by 168
  15. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 4; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 169
  2. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  3. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  4. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 95
  5. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 87
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  9. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 38
  10. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 37
  11. 2024Immunization against harmful fine-tuning attackscited by 35
  12. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Modelscited by 32
11 more
  1. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 29
  2. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 20
  3. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguardscited by 17
  4. 2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
  5. 2024Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teamingcited by 10
  6. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  7. 2026Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defensecited by 3
  8. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  9. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  10. 2026On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignmentcited by 1
  11. 2025EnchTable: Unified Safety Alignment Transfer in Fine-tuned Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Automated Progressive Red Teaming
  2. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
  3. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
  4. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
  5. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
  6. 2025Multi-lingual Multi-turn Automated Red Teaming for LLMs
  7. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding
  8. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models