# CoNTACT: A Dutch COVID-19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection

arXiv: https://arxiv.org/abs/2203.07362 · HTML: https://mlsanity.com/paper/2203.07362/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [Self-supervised Domain Adaptation for Computer Vision Tasks](https://mlsanity.com/paper/1907.10915/) (2019)
- [How transfer learning impacts linguistic knowledge in deep NLP models?](https://mlsanity.com/paper/2105.15179/) (2021)
- [Automatic Detection of COVID-19 Vaccine Misinformation with Graph Link Prediction](https://mlsanity.com/paper/2108.02314/) (2021)
- [Domain Adaptation for Sentiment Analysis Using Increased Intraclass Separation](https://mlsanity.com/paper/2107.01598/) (2021)
- [Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling](https://mlsanity.com/paper/1904.02817/) (2019)

## Built on by (most cited first, 2 of 2)

- [RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use](https://mlsanity.com/paper/2211.08192/) (2022)
- [Bag of Lies: Robustness in Continuous Pre-training BERT](https://mlsanity.com/paper/2406.09967/) (2024)

## Abstract (arXiv)

We present CoNTACT: a Dutch language model adapted to the domain of COVID-19 tweets. The model was developed by continuing the pre-training phase of RobBERT (Delobelle, 2020) by using 2.8M Dutch COVID-19 related tweets posted in 2021. In order to test the performance of the model and compare it to RobBERT, the two models were tested on two tasks: (1) binary vaccine hesitancy detection and (2) detection of arguments for vaccine hesitancy. For both tasks, not only Twitter but also Facebook data was used to show cross-genre performance. In our experiments, CoNTACT showed statistically significant gains over RobBERT in all experiments for task 1. For task 2, we observed substantial improvements in virtually all classes in all experiments. An error analysis indicated that the domain adaptation yielded better representations of domain-specific terminology, causing CoNTACT to make more accurate classification decisions.