# UHH-LT at SemEval-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection

arXiv: https://arxiv.org/abs/2004.11493 · HTML: https://mlsanity.com/paper/2004.11493/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2020.

## Built on by (most cited first, 9 of 9)

- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- [MUDES: Multilingual Detection of Offensive Spans](https://mlsanity.com/paper/2102.09665/) (2021)
- [A systematic review of Hate Speech automatic detection using Natural Language Processing](https://mlsanity.com/paper/2106.00742/) (2021)
- [Emotion Based Hate Speech Detection using Multimodal Learning](https://mlsanity.com/paper/2202.06218/) (2022)
- [ToxCCIn: Toxic Content Classification with Interpretability](https://mlsanity.com/paper/2103.01328/) (2021)
- [Using LLMs to discover emerging coded antisemitic hate-speech in extremist social media](https://mlsanity.com/paper/2401.10841/) (2024)
- [HLE-UPC at SemEval-2021 Task 5: Multi-Depth DistilBERT for Toxic Spans Detection](https://mlsanity.com/paper/2104.00639/) (2021)
- [UoB at SemEval-2021 Task 5: Extending Pre-Trained Language Models to Include Task and Domain-Specific Information for Toxic Span Prediction](https://mlsanity.com/paper/2110.03730/) (2021)
- [Neural Models for Offensive Language Detection](https://mlsanity.com/paper/2106.14609/) (2021)

## Abstract (arXiv)

Fine-tuning of pre-trained transformer networks such as BERT yield state-of-the-art results for text classification tasks. Typically, fine-tuning is performed on task-specific training datasets in a supervised manner. One can also fine-tune in unsupervised manner beforehand by further pre-training the masked language modeling (MLM) task. Hereby, in-domain data for unsupervised MLM resembling the actual classification target dataset allows for domain adaptation of the model. In this paper, we compare current pre-trained transformer networks with and without MLM fine-tuning on their performance for offensive language detection. Our MLM fine-tuned RoBERTa-based classifier officially ranks 1st in the SemEval 2020 Shared Task~12 for the English language. Further experiments with the ALBERT model even surpass this result.