# Task Adaptive Pretraining of Transformers for Hostility Detection

arXiv: https://arxiv.org/abs/2101.03382 · HTML: https://mlsanity.com/paper/2101.03382/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 9 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [Spread of hate speech in online social media](https://mlsanity.com/paper/1812.01693/) (2018)
- [Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizations](https://mlsanity.com/paper/2001.05495/) (2020)
- [Hostility Detection Dataset in Hindi](https://mlsanity.com/paper/2011.03588/) (2020)
- [emoji2vec: Learning Emoji Representations from their Description](https://mlsanity.com/paper/1609.08359/) (2016)
- [Leveraging Multilingual Transformers for Hate Speech Detection](https://mlsanity.com/paper/2101.03207/) (2021)

## Built on by (most cited first, 1 of 1)

- [Offense Detection in Dravidian Languages using Code-Mixing Index based Focal Loss](https://mlsanity.com/paper/2111.06916/) (2021)

## Abstract (arXiv)

Identifying adverse and hostile content on the web and more particularly, on social media, has become a problem of paramount interest in recent years. With their ever increasing popularity, fine-tuning of pretrained Transformer-based encoder models with a classifier head are gradually becoming the new baseline for natural language classification tasks. In our work, we explore the gains attributed to Task Adaptive Pretraining (TAPT) prior to fine-tuning of Transformer-based architectures. We specifically study two problems, namely, (a) Coarse binary classification of Hindi Tweets into Hostile or Not, and (b) Fine-grained multi-label classification of Tweets into four categories: hate, fake, offensive, and defamation. Building up on an architecture which takes emojis and segmented hashtags into consideration for classification, we are able to experimentally showcase the performance upgrades due to TAPT. Our system (with team name 'iREL IIIT') ranked first in the 'Hostile Post Detection in Hindi' shared task with an F1 score of 97.16% for coarse-grained detection and a weighted F1 score of 62.96% for fine-grained multi-label classification on the provided blind test corpora.