mlsanityData to 14 Oct 2026

Robust Conversational Agents against Imperceptible Toxicity Triggers

arXiv 2205.02392 ↗First posted 5 May 2022cs.CL cs.AIcited by 23

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Warning: this paper contains content that maybe offensive or upsetting. Recent research in Natural Language Processing (NLP) has advanced the development of various toxicity detection models with the intention of identifying and mitigating toxic language from existing systems. Despite the abundance of research in this area, less attention has been given to adversarial attacks that force the system to generate toxic language and the defense against them. Existing work to generate such attacks is either based on human-generated attacks which is costly and not scalable or, in case of automatic attacks, the attack vector does not conform to human-like language, which can be detected using a language model loss. In this work, we propose attacks against conversational agents that are imperceptible, i.e., they fit the conversation in terms of coherency, relevancy, and fluency, while they are effective and scalable, i.e., they can automatically trigger the system into generating toxic language. We then propose a defense mechanism against such attacks which not only mitigates the attack but also attempts to maintain the conversational flow. Through automatic and human evaluations, we show that our defense is effective at avoiding toxic language generation even against imperceptible toxicity triggers while the generated language fits the conversation in terms of coherency and relevancy. Lastly, we establish the generalizability of such a defense mechanism on language generation models beyond conversational agents.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 31 references that are in the map. First posted on arXiv 5 May 2022.

Builds on

22 of 31 references.

References · 22

  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  5. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  7. 2018Adversarial Attacks and Defences: A Surveycited by 259
  8. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  9. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  10. 2020Towards Controllable Biases in Language Generationcited by 112
  11. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  12. 2020Reevaluating Adversarial Examples in Natural Languagecited by 85
10 more
  1. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 78
  2. 2018Conversations Gone Awry: Detecting Early Signs of Conversational Failurecited by 73
  3. 2020Poisoning Attacks on Algorithmic Fairnesscited by 59
  4. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  5. 2020Exacerbating Algorithmic Bias through Fairness Attackscited by 47
  6. 2019Word-level Textual Adversarial Attacking as Combinatorial Optimizationcited by 37
  7. 2018Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Modelscited by 26
  8. 2021Attributing Fair Decisions with Attention Interventionscited by 10
  9. 2021Local Explanation of Dialogue Response Generationcited by 7
  10. 2021Contrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classificationcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 9; 2024: 9; 2025: 4.

23 papers.

Sort:

By area:

  1. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  2. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  3. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
  4. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 59
  5. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  6. 2024ESCoT: Towards Interpretable Emotional Support Dialogue Systemscited by 46
  7. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
  8. 2024Gradient-Based Language Model Red Teamingcited by 34
  9. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 33
  10. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 25
  11. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14
  12. 2023Learn What NOT to Learn: Towards Generative Safety in Chatbotscited by 8
11 more
  1. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  2. 2023Prompts have evil twinscited by 7
  3. 2023Run Like a Girl! Sports-Related Gender Bias in Language and Visioncited by 6
  4. 2024Decoding Hate: Exploring Language Models' Reactions to Hate Speechcited by 6
  5. 2023Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attackscited by 5
  6. 2023JAB: Joint Adversarial Prompting and Belief Augmentationcited by 5
  7. 2025Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Modelscited by 4
  8. 2025EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Modelscited by 2
  9. 2024Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Searchcited by 1
  10. 2024Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languagescited by 0
  11. 2025RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AI
  2. 2026Measuring and Mitigating Toxicity in Large Language Models: A Comprehensive Replication Study
  3. 2025Redefining Experts: Interpretable Decomposition of Language Models for Toxicity Mitigation
  4. 2026Evaluating Language Model Safety Across Long Adversarial Conversations
  5. 2024Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models
  6. 2023Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks
  7. 2024Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents
  8. 2025Attack and defense techniques in large language models: A survey and new perspectives