# Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls

arXiv: https://arxiv.org/abs/2208.03295 · HTML: https://mlsanity.com/paper/2208.03295/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 5 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Continual Learning in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2012.15504/) (2020)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [Deep Bilevel Learning](https://mlsanity.com/paper/1809.01465/) (2018)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions](https://mlsanity.com/paper/2006.12442/) (2020)
- [A Survey of Available Corpora for Building Data-Driven Dialogue Systems](https://mlsanity.com/paper/1512.05742/) (2015)
- [Deploying Lifelong Open-Domain Dialogue Learning](https://mlsanity.com/paper/2008.08076/) (2020)
- [A Survey of NLP-Related Crowdsourcing HITs: what works and what does not](https://mlsanity.com/paper/2111.05241/) (2021)
- [Hunting for Troll Comments in News Community Forums](https://mlsanity.com/paper/1911.08113/) (2019)

## Built on by (most cited first, 11 of 11)

- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [The CRINGE Loss: Learning what language not to model](https://mlsanity.com/paper/2211.05826/) (2022)
- [Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models](https://mlsanity.com/paper/2407.12824/) (2024)
- [Learning to Intervene on Concept Bottlenecks](https://mlsanity.com/paper/2308.13453/) (2023)
- [When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels](https://mlsanity.com/paper/2210.15893/) (2022)
- [Leveraging Implicit Feedback from Deployment Data in Dialogue](https://mlsanity.com/paper/2307.14117/) (2023)
- [Towards Boosting the Open-Domain Chatbot with Human Feedback](https://mlsanity.com/paper/2208.14165/) (2022)
- [Improving Open Language Models by Learning from Organic Interactions](https://mlsanity.com/paper/2306.04707/) (2023)
- [Revision Transformers: Instructing Language Models to Change their Values](https://mlsanity.com/paper/2210.10332/) (2022)
- [Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task](https://mlsanity.com/paper/2310.19271/) (2023)

## Abstract (arXiv)

The promise of interaction between intelligent conversational agents and humans is that models can learn from such feedback in order to improve. Unfortunately, such exchanges in the wild will not always involve human utterances that are benign or of high quality, and will include a mixture of engaged (helpers) and unengaged or even malicious users (trolls). In this work we study how to perform robust learning in such an environment. We introduce a benchmark evaluation, SafetyMix, which can evaluate methods that learn safe vs. toxic language in a variety of adversarial settings to test their robustness. We propose and analyze several mitigating learning algorithms that identify trolls either at the example or at the user level. Our main finding is that user-based methods, that take into account that troll users will exhibit adversarial behavior across multiple examples, work best in a variety of settings on our benchmark. We then test these methods in a further real-life setting of conversations collected during deployment, with similar results.