# FH-SWF SG at GermEval 2021: Using Transformer-Based Language Models to Identify Toxic, Engaging, & Fact-Claiming Comments

arXiv: https://arxiv.org/abs/2109.02966 · HTML: https://mlsanity.com/paper/2109.02966/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 64 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection](https://mlsanity.com/paper/1809.08193/) (2018)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020)
- [Which *BERT? A Survey Organizing Contextualized Encoders](https://mlsanity.com/paper/2010.00854/) (2020)
- [Top Comment or Flop Comment? Predicting and Explaining User Engagement in Online News Discussions](https://mlsanity.com/paper/2003.11949/) (2020)

## Abstract (arXiv)

In this paper we describe the methods we used for our submissions to the GermEval 2021 shared task on the identification of toxic, engaging, and fact-claiming comments. For all three subtasks we fine-tuned freely available transformer-based models from the Huggingface model hub. We evaluated the performance of various pre-trained models after fine-tuning on 80% of the training data with different hyperparameters and submitted predictions of the two best performing resulting models. We found that this approach worked best for subtask 3, for which we achieved an F1-score of 0.736.