# No Offense Taken: Eliciting Offensiveness from Language Models

arXiv: https://arxiv.org/abs/2310.00892 · HTML: https://mlsanity.com/paper/2310.00892/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation](https://mlsanity.com/paper/2104.08678/) (2021)
- [The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents](https://mlsanity.com/paper/1911.03768/) (2019)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)

## Built on by (most cited first, 3 of 3)

- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations](https://mlsanity.com/paper/2410.09097/) (2024)
- [ToxSearch: Evolving Prompts for Toxicity Search in Large Language Models](https://mlsanity.com/paper/2511.12487/) (2025)

## Abstract (arXiv)

This work was completed in May 2022. For safe and reliable deployment of language models in the real world, testing needs to be robust. This robustness can be characterized by the difficulty and diversity of the test cases we evaluate these models on. Limitations in human-in-the-loop test case generation has prompted an advent of automated test case generation approaches. In particular, we focus on Red Teaming Language Models with Language Models by Perez et al.(2022). Our contributions include developing a pipeline for automated test case generation via red teaming that leverages publicly available smaller language models (LMs), experimenting with different target LMs and red classifiers, and generating a corpus of test cases that can help in eliciting offensive responses from widely deployed LMs and identifying their failure modes.