# Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content

arXiv: https://arxiv.org/abs/2308.13768 · HTML: https://mlsanity.com/paper/2308.13768/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 45 of its 52 references that are in the map.
- First posted on arXiv 26 Aug 2023.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 45)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [The Science of Detecting LLM-Generated Texts](https://mlsanity.com/paper/2303.07205/) (2023)
- [Does Gender Matter? Towards Fairness in Dialogue Systems](https://mlsanity.com/paper/1910.10486/) (2019)
- [BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT](https://mlsanity.com/paper/2304.12298/) (2023)
- [Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA](https://mlsanity.com/paper/1906.07132/) (2019)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Training Language Models with Language Feedback](https://mlsanity.com/paper/2204.14146/) (2022)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)

## Built on by (most cited first, 7 of 7)

- [LLM-based Multi-Agent Systems: Techniques and Business Perspectives](https://mlsanity.com/paper/2411.14033/) (2024)
- [Single-pass Detection of Jailbreaking Input in Large Language Models](https://mlsanity.com/paper/2502.15435/) (2025)
- [Shh, don't say that! Domain Certification in LLMs](https://mlsanity.com/paper/2502.19320/) (2025)
- [Adversarial Preference Learning for Robust LLM Alignment](https://mlsanity.com/paper/2505.24369/) (2025)
- [OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!](https://mlsanity.com/paper/2509.26495/) (2025)
- [Attesting Model Lineage by Consisted Knowledge Evolution with Fine-Tuning Trajectory](https://mlsanity.com/paper/2601.11683/) (2026)
- [TombRaider: Entering the Vault of History to Jailbreak Large Language Models](https://mlsanity.com/paper/2501.18628/) (2025)

## Abstract (arXiv)

In this paper, we tackle the emerging challenge of unintended harmful content generation in Large Language Models (LLMs) with a novel dual-stage optimisation technique using adversarial fine-tuning. Our two-pronged approach employs an adversarial model, fine-tuned to generate potentially harmful prompts, and a judge model, iteratively optimised to discern these prompts. In this adversarial cycle, the two models seek to outperform each other in the prompting phase, generating a dataset of rich examples which are then used for fine-tuning. This iterative application of prompting and fine-tuning allows continuous refinement and improved performance. The performance of our approach is evaluated through classification accuracy on a dataset consisting of problematic prompts not detected by GPT-4, as well as a selection of contentious but unproblematic prompts. We show considerable increase in classification accuracy of the judge model on this challenging dataset as it undergoes the optimisation process. Furthermore, we show that a rudimentary model textttada can achieve 13% higher accuracy on the hold-out test set than GPT-4 after only a few rounds of this process, and that this fine-tuning improves performance in parallel tasks such as toxic comment identification.