# Robust Conversational Agents against Imperceptible Toxicity Triggers arXiv: https://arxiv.org/abs/2205.02392 ยท HTML: https://mlsanity.com/paper/2205.02392/ ## Key facts - Cited by 23 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 22 of its 31 references that are in the map. - First posted on arXiv 5 May 2022. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 22 of 22) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020) - [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019) - [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019) - [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019) - [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018) - [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021) - [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018) - [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019) - [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020) - [Towards Controllable Biases in Language Generation](https://mlsanity.com/paper/2005.00268/) (2020) - [Ethical Challenges in Data-Driven Dialogue Systems](https://mlsanity.com/paper/1711.09050/) (2017) - [Reevaluating Adversarial Examples in Natural Language](https://mlsanity.com/paper/2004.14174/) (2020) - [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021) - [Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://mlsanity.com/paper/1805.05345/) (2018) - [Poisoning Attacks on Algorithmic Fairness](https://mlsanity.com/paper/2004.07401/) (2020) - [Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts](https://mlsanity.com/paper/2108.11830/) (2021) - [Exacerbating Algorithmic Bias through Fairness Attacks](https://mlsanity.com/paper/2012.08723/) (2020) - [Word-level Textual Adversarial Attacking as Combinatorial Optimization](https://mlsanity.com/paper/1910.12196/) (2019) - [Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models](https://mlsanity.com/paper/1809.02079/) (2018) - [Attributing Fair Decisions with Attention Interventions](https://mlsanity.com/paper/2109.03952/) (2021) - [Local Explanation of Dialogue Response Generation](https://mlsanity.com/paper/2106.06528/) (2021) - [Contrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classification](https://mlsanity.com/paper/2109.04385/) (2021) ## Built on by (most cited first, 23 of 23) - [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023) - [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023) - [FLIRT: Feedback Loop In-context Red Teaming](https://mlsanity.com/paper/2308.04265/) (2023) - [White-box Multimodal Jailbreaks Against Large Vision-Language Models](https://mlsanity.com/paper/2405.17894/) (2024) - [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025) - [ESCoT: Towards Interpretable Emotional Support Dialogue Systems](https://mlsanity.com/paper/2406.10960/) (2024) - [Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment](https://mlsanity.com/paper/2411.18688/) (2024) - [Gradient-Based Language Model Red Teaming](https://mlsanity.com/paper/2401.16656/) (2024) - [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022) - [Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning](https://mlsanity.com/paper/2412.18693/) (2024) - [Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements](https://mlsanity.com/paper/2302.09270/) (2023) - [Learn What NOT to Learn: Towards Generative Safety in Chatbots](https://mlsanity.com/paper/2304.11220/) (2023) - [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024) - [Prompts have evil twins](https://mlsanity.com/paper/2311.07064/) (2023) - [Run Like a Girl! Sports-Related Gender Bias in Language and Vision](https://mlsanity.com/paper/2305.14468/) (2023) - [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024) - [Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks](https://mlsanity.com/paper/2310.16955/) (2023) - [JAB: Joint Adversarial Prompting and Belief Augmentation](https://mlsanity.com/paper/2311.09473/) (2023) - [Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models](https://mlsanity.com/paper/2508.01741/) (2025) - [EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models](https://mlsanity.com/paper/2502.14976/) (2025) - [Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search](https://mlsanity.com/paper/2408.08899/) (2024) - [Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languages](https://mlsanity.com/paper/2412.10805/) (2024) - [RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusion](https://mlsanity.com/paper/2503.06223/) (2025) ## Abstract (arXiv) Warning: this paper contains content that maybe offensive or upsetting. Recent research in Natural Language Processing (NLP) has advanced the development of various toxicity detection models with the intention of identifying and mitigating toxic language from existing systems. Despite the abundance of research in this area, less attention has been given to adversarial attacks that force the system to generate toxic language and the defense against them. Existing work to generate such attacks is either based on human-generated attacks which is costly and not scalable or, in case of automatic attacks, the attack vector does not conform to human-like language, which can be detected using a language model loss. In this work, we propose attacks against conversational agents that are imperceptible, i.e., they fit the conversation in terms of coherency, relevancy, and fluency, while they are effective and scalable, i.e., they can automatically trigger the system into generating toxic language. We then propose a defense mechanism against such attacks which not only mitigates the attack but also attempts to maintain the conversational flow. Through automatic and human evaluations, we show that our defense is effective at avoiding toxic language generation even against imperceptible toxicity triggers while the generated language fits the conversation in terms of coherency and relevancy. Lastly, we establish the generalizability of such a defense mechanism on language generation models beyond conversational agents.