# Predicting Different Types of Subtle Toxicity in Unhealthy Online Conversations

arXiv: https://arxiv.org/abs/2106.03952 · HTML: https://mlsanity.com/paper/2106.03952/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 69 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Challenges for Toxic Comment Classification: An In-Depth Error Analysis](https://mlsanity.com/paper/1809.07572/) (2018)
- [A Just and Comprehensive Strategy for Using NLP to Address Online Abuse](https://mlsanity.com/paper/1906.01738/) (2019)
- [Data Augmentation via Dependency Tree Morphing for Low-Resource Languages](https://mlsanity.com/paper/1903.09460/) (2019)
- [Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue](https://mlsanity.com/paper/1709.05404/) (2017)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [A Network Perspective on Stratification of Multi-Label Data](https://mlsanity.com/paper/1704.08756/) (2017)
- [A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1610.08815/) (2016)
- [Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula](https://mlsanity.com/paper/1606.04142/) (2016)
- [CASCADE: Contextual Sarcasm Detection in Online Discussion Forums](https://mlsanity.com/paper/1805.06413/) (2018)
- [A scikit-based Python environment for performing multi-label classification](https://mlsanity.com/paper/1702.01460/) (2017)
- [Convolutional Neural Networks for Toxic Comment Classification](https://mlsanity.com/paper/1802.09957/) (2018)
- [TalkDown: A Corpus for Condescension Detection in Context](https://mlsanity.com/paper/1909.11272/) (2019)
- [Six Attributes of Unhealthy Conversation](https://mlsanity.com/paper/2010.07410/) (2020)
- [Don't Patronize Me! An Annotated Dataset with Patronizing and Condescending Language towards Vulnerable Communities](https://mlsanity.com/paper/2011.08320/) (2020)
- [Forecasting the presence and intensity of hostility on Instagram using linguistic and social features](https://mlsanity.com/paper/1804.06759/) (2018)
- [GPU-acceleration for Large-scale Tree Boosting](https://mlsanity.com/paper/1706.08359/) (2017)
- [Sarcasm Detection using Context Separators in Online Discourse](https://mlsanity.com/paper/2006.00850/) (2020)

## Built on by (most cited first, 5 of 5)

- [Toxicity Classification in Ukrainian](https://mlsanity.com/paper/2404.17841/) (2024)
- [ML_LTU at SemEval-2022 Task 4: T5 Towards Identifying Patronizing and Condescending Language](https://mlsanity.com/paper/2204.07432/) (2022)
- [Position: Evaluation Scores Are Perishable Knowledge Claims](https://mlsanity.com/paper/2607.26191/) (2026)
- [How We Define Harm Impacts Data Annotations: Explaining How Annotators Distinguish Hateful, Offensive, and Toxic Comments](https://mlsanity.com/paper/2309.15827/) (2023)
- [Quantifying Feature Importance for Online Content Moderation](https://mlsanity.com/paper/2510.19882/) (2025)

## Abstract (arXiv)

This paper investigates the use of machine learning models for the classification of unhealthy online conversations containing one or more forms of subtler abuse, such as hostility, sarcasm, and generalization. We leveraged a public dataset of 44K online comments containing healthy and unhealthy comments labeled with seven forms of subtle toxicity. We were able to distinguish between these comments with a top micro F1-score, macro F1-score, and ROC-AUC of 88.76%, 67.98%, and 0.71, respectively. Hostile comments were easier to detect than other types of unhealthy comments. We also conducted a sentiment analysis which revealed that most types of unhealthy comments were associated with a slight negative sentiment, with hostile comments being the most negative ones.