# Investigating label suggestions for opinion mining in German Covid-19 social media

arXiv: https://arxiv.org/abs/2105.12980 · HTML: https://mlsanity.com/paper/2105.12980/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours](https://mlsanity.com/paper/1704.05972/) (2017)
- [From Arguments to Key Points: Towards Automatic Argument Summarization](https://mlsanity.com/paper/2005.01619/) (2020)
- [Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter](https://mlsanity.com/paper/2005.00388/) (2020)
- [A Retrospective Analysis of the Fake News Challenge Stance Detection Task](https://mlsanity.com/paper/1806.05180/) (2018)
- [Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages](https://mlsanity.com/paper/1605.05894/) (2016)
- [Integrating Stance Detection and Fact Checking in a Unified Corpus](https://mlsanity.com/paper/1804.08012/) (2018)
- [Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains](https://mlsanity.com/paper/1906.02564/) (2019)

## Built on by (most cited first, 15 of 15)

- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [Hierarchical and Incremental Structural Entropy Minimization for Unsupervised Social Event Detection](https://mlsanity.com/paper/2312.11891/) (2023)
- [Automatic Identification and Classification of Bragging in Social Media](https://mlsanity.com/paper/2203.05840/) (2022)
- [Annotation Curricula to Implicitly Train Non-Expert Annotators](https://mlsanity.com/paper/2106.02382/) (2021)
- [AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters](https://mlsanity.com/paper/2108.08103/) (2021)
- [A Hybrid Intelligence Method for Argument Mining](https://mlsanity.com/paper/2403.09713/) (2024)
- [Unsupervised Extractive Summarization of Emotion Triggers](https://mlsanity.com/paper/2306.01444/) (2023)
- [Emotion analysis and detection during COVID-19](https://mlsanity.com/paper/2107.11020/) (2021)
- [RuArg-2022: Argument Mining Evaluation](https://mlsanity.com/paper/2206.09249/) (2022)
- [ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentation](https://mlsanity.com/paper/2505.22076/) (2025)
- [TexPrax: A Messaging Application for Ethical, Real-time Data Collection and Annotation](https://mlsanity.com/paper/2208.07846/) (2022)
- [Data and models for stance and premise detection in COVID-19 tweets: insights from the Social Media Mining for Health (SMM4H) 2022 shared task](https://mlsanity.com/paper/2311.08057/) (2023)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Adaptive Differentially Private Structural Entropy Minimization for Unsupervised Social Event Detection](https://mlsanity.com/paper/2407.18274/) (2024)
- [Contextualized Prompting For Stance Detection On Social Media](https://mlsanity.com/paper/2606.06022/) (2026)

## Abstract (arXiv)

This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement - in terms of inter-annotator agreement(+.14 Fleiss' kappa) and annotation quality - compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups. To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations.