# MIPT-NSU-UTMN at SemEval-2021 Task 5: Ensembling Learning with Pre-trained Language Models for Toxic Spans Detection

arXiv: https://arxiv.org/abs/2104.04739 · HTML: https://mlsanity.com/paper/2104.04739/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 10 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media](https://mlsanity.com/paper/1910.12574/) (2019)
- [Offensive Language Identification in Greek](https://mlsanity.com/paper/2003.07459/) (2020)
- [Deep Learning Models for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2004.06465/) (2020)
- [One-step and Two-step Classification for Abusive Language Detection on Twitter](https://mlsanity.com/paper/1706.01206/) (2017)
- [Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis](https://mlsanity.com/paper/2010.04543/) (2020)
- [Deep Learning for User Comment Moderation](https://mlsanity.com/paper/1705.09993/) (2017)
- [Comparative Studies of Detecting Abusive Language on Twitter](https://mlsanity.com/paper/1808.10245/) (2018)
- [Context-Aware Attention for Understanding Twitter Abuse](https://mlsanity.com/paper/1809.08726/) (2018)

## Built on by (most cited first, 1 of 1)

- [ViHOS: Hate Speech Spans Detection for Vietnamese](https://mlsanity.com/paper/2301.10186/) (2023)

## Abstract (arXiv)

This paper describes our system for SemEval-2021 Task 5 on Toxic Spans Detection. We developed ensemble models using BERT-based neural architectures and post-processing to combine tokens into spans. We evaluated several pre-trained language models using various ensemble techniques for toxic span identification and achieved sizable improvements over our baseline fine-tuned BERT models. Finally, our system obtained a F1-score of 67.55% on test data.