# AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detection

arXiv: https://arxiv.org/abs/2212.10405 · HTML: https://mlsanity.com/paper/2212.10405/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 64 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- [A Just and Comprehensive Strategy for Using NLP to Address Online Abuse](https://mlsanity.com/paper/1906.01738/) (2019)
- [Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking](https://mlsanity.com/paper/1904.06707/) (2019)
- [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021)
- [Like trainer, like bot? Inheritance of bias in algorithmic content moderation](https://mlsanity.com/paper/1707.01477/) (2017)
- [Learning from Crowds by Modeling Common Confusions](https://mlsanity.com/paper/2012.13052/) (2020)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [GILE: A Generalized Input-Label Embedding for Text Classification](https://mlsanity.com/paper/1806.06219/) (2018)
- [GraphNLI: A Graph-based Natural Language Inference Model for Polarity Prediction in Online Debates](https://mlsanity.com/paper/2202.08175/) (2022)
- [A Graph-Based Context-Aware Model to Understand Online Conversations](https://mlsanity.com/paper/2211.09207/) (2022)
- [Cross-lingual Capsule Network for Hate Speech Detection in Social Media](https://mlsanity.com/paper/2108.03089/) (2021)

## Built on by (most cited first, 12 of 12)

- [Causality Guided Disentanglement for Cross-Platform Hate Speech Detection](https://mlsanity.com/paper/2308.02080/) (2023)
- [Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation](https://mlsanity.com/paper/2410.23448/) (2024)
- [PEACE: Cross-Platform Hate Speech Detection- A Causality-guided Framework](https://mlsanity.com/paper/2306.08804/) (2023)
- [Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology](https://mlsanity.com/paper/2406.14099/) (2024)
- [A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement](https://mlsanity.com/paper/2411.04090/) (2024)
- [Protecting Vulnerable Voices: Synthetic Dataset Generation for Self-Disclosure Detection](https://mlsanity.com/paper/2507.22930/) (2025)
- [Detecting and Preventing Harmful Behaviors in AI Companions: Development and Evaluation of the SHIELD Supervisory System](https://mlsanity.com/paper/2510.15891/) (2025)
- [Cross-Platform Hate Speech Detection with Weakly Supervised Causal Disentanglement](https://mlsanity.com/paper/2404.11036/) (2024)
- [Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models](https://mlsanity.com/paper/2506.08593/) (2025)
- [Multiclass Hate Speech Detection with RoBERTa-OTA: Integrating Transformer Attention and Graph Convolutional Networks](https://mlsanity.com/paper/2603.04414/) (2026)
- [Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks](https://mlsanity.com/paper/2605.09955/) (2026)
- [When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detection](https://mlsanity.com/paper/2605.27313/) (2026)

## Abstract (arXiv)

Supervised approaches generally rely on majority-based labels. However, it is hard to achieve high agreement among annotators in subjective tasks such as hate speech detection. Existing neural network models principally regard labels as categorical variables, while ignoring the semantic information in diverse label texts. In this paper, we propose AnnoBERT, a first-of-its-kind architecture integrating annotator characteristics and label text with a transformer-based model to detect hate speech, with unique representations based on each annotator's characteristics via Collaborative Topic Regression (CTR) and integrate label text to enrich textual representations. During training, the model associates annotators with their label choices given a piece of text; during evaluation, when label information is not available, the model predicts the aggregated label given by the participating annotators by utilising the learnt association. The proposed approach displayed an advantage in detecting hate speech, especially in the minority class and edge cases with annotator disagreement. Improvement in the overall performance is the largest when the dataset is more label-imbalanced, suggesting its practical value in identifying real-world hate speech, as the volume of hate speech in-the-wild is extremely small on social media, when compared with normal (non-hate) speech. Through ablation studies, we show the relative contributions of annotator embeddings and label text to the model performance, and tested a range of alternative annotator embeddings and label text combinations.