# FBERT: A Neural Transformer for Identifying Offensive Content

arXiv: https://arxiv.org/abs/2109.05074 · HTML: https://mlsanity.com/paper/2109.05074/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification](https://mlsanity.com/paper/2004.14454/) (2020)
- [Detecting Hate Speech in Social Media](https://mlsanity.com/paper/1712.06427/) (2017)
- [MUDES: Multilingual Detection of Offensive Spans](https://mlsanity.com/paper/2102.09665/) (2021)

## Built on by (most cited first, 30 of 30)

- [HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning](https://mlsanity.com/paper/2311.00321/) (2023)
- [ToViLaG: Your Visual-Language Generative Model is Also An Evildoer](https://mlsanity.com/paper/2312.11523/) (2023)
- [Offensive Language Identification in Transliterated and Code-Mixed Bangla](https://mlsanity.com/paper/2311.15023/) (2023)
- [NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data](https://mlsanity.com/paper/2403.19260/) (2024)
- [Character-level HyperNetworks for Hate Speech Detection](https://mlsanity.com/paper/2111.06336/) (2021)
- ["Define Your Terms" : Enhancing Efficient Offensive Speech Classification with Definition](https://mlsanity.com/paper/2402.03221/) (2024)
- [DefVerify: Do Hate Speech Models Reflect Their Dataset's Definition?](https://mlsanity.com/paper/2410.15911/) (2024)
- [BERT(s) to Detect Multiword Expressions](https://mlsanity.com/paper/2208.07832/) (2022)
- [Overview of the HASOC Subtrack at FIRE 2022: Offensive Language Identification in Marathi](https://mlsanity.com/paper/2211.10163/) (2022)
- [Predicting the Type and Target of Offensive Social Media Posts in Marathi](https://mlsanity.com/paper/2211.12570/) (2022)
- [OffMix-3L: A Novel Code-Mixed Dataset in Bangla-English-Hindi for Offensive Language Identification](https://mlsanity.com/paper/2310.18387/) (2023)
- [HateCOT: An Explanation-Enhanced Dataset for Generalizable Offensive Speech Detection via Large Language Models](https://mlsanity.com/paper/2403.11456/) (2024)
- [Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection](https://mlsanity.com/paper/2210.04267/) (2022)
- [Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi](https://mlsanity.com/paper/2309.10272/) (2023)
- [A Federated Learning Approach to Privacy Preserving Offensive Language Identification](https://mlsanity.com/paper/2404.11470/) (2024)
- [Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection](https://mlsanity.com/paper/2406.07886/) (2024)
- [A Twitter BERT Approach for Offensive Language Detection in Marathi](https://mlsanity.com/paper/2212.10039/) (2022)
- [A Text-to-Text Model for Multilingual Offensive Language Identification](https://mlsanity.com/paper/2312.03379/) (2023)
- [MasonPerplexity at ClimateActivism 2024: Integrating Advanced Ensemble Techniques and Data Augmentation for Climate Activism Stance and Hate Event Identification](https://mlsanity.com/paper/2402.01976/) (2024)
- [ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model](https://mlsanity.com/paper/2405.14141/) (2024)
- [SOLD: Sinhala Offensive Language Dataset](https://mlsanity.com/paper/2212.00851/) (2022)
- [Trustworthy Hate Speech Detection Through Visual Augmentation](https://mlsanity.com/paper/2409.13557/) (2024)
- [A Federated Approach to Few-Shot Hate Speech Detection for Marginalized Communities](https://mlsanity.com/paper/2412.04942/) (2024)
- [Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge](https://mlsanity.com/paper/2311.14865/) (2023)
- [Towards Generalized Offensive Language Identification](https://mlsanity.com/paper/2407.18738/) (2024)
- [MatchLM2Lite: A Scalable MLLM-to-Lite Framework for Reproduced Content Identification](https://mlsanity.com/paper/2606.14786/) (2026)
- [SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Models](https://mlsanity.com/paper/2509.15174/) (2025)
- [Exploring the Performance of Large Language Models on Subjective Span Identification Tasks](https://mlsanity.com/paper/2601.00736/) (2026)
- [ToxiTrace: Gradient-Aligned Training for Explainable Chinese Toxicity Detection](https://mlsanity.com/paper/2604.12321/) (2026)
- [Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderation](https://mlsanity.com/paper/2607.28187/) (2026)

## Abstract (arXiv)

Transformer-based models such as BERT, XLNET, and XLM-R have achieved state-of-the-art performance across various NLP tasks including the identification of offensive language and hate speech, an important problem in social media. In this paper, we present fBERT, a BERT model retrained on SOLID, the largest English offensive language identification corpus available with over 1.4 million offensive instances. We evaluate fBERT's performance on identifying offensive content on multiple English datasets and we test several thresholds for selecting instances from SOLID. The fBERT model will be made freely available to the community.