# Towards A Reliable Ground-Truth For Biased Language Detection

arXiv: https://arxiv.org/abs/2112.07421 · HTML: https://mlsanity.com/paper/2112.07421/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 10 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 2 of 2)

- [MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristics](https://mlsanity.com/paper/2105.11910/) (2021)
- [Neural Based Statement Classification for Biased Language](https://mlsanity.com/paper/1811.05740/) (2018)

## Built on by (most cited first, 10 of 10)

- [Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection](https://mlsanity.com/paper/2304.13148/) (2023)
- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection](https://mlsanity.com/paper/2411.11081/) (2024)
- [Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles](https://mlsanity.com/paper/2211.03491/) (2022)
- [The FIGNEWS Shared Task on News Media Narratives](https://mlsanity.com/paper/2407.18147/) (2024)
- [An Interdisciplinary Approach for the Automated Detection and Visualization of Media Bias in News Articles](https://mlsanity.com/paper/2112.13352/) (2021)
- [MAGPIE: Multi-Task Media-Bias Analysis Generalization for Pre-Trained Identification of Expressions](https://mlsanity.com/paper/2403.07910/) (2024)
- [Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Dataset](https://mlsanity.com/paper/2407.13896/) (2024)
- [Code Book for the Annotation of Diverse Cross-Document Coreference of Entities in News Articles](https://mlsanity.com/paper/2310.12064/) (2023)
- [MABPD: Multi-Agent Bias Probing & Detection via Structured Argument Debate](https://mlsanity.com/paper/2609.04841/) (2026)

## Abstract (arXiv)

Reference texts such as encyclopedias and news articles can manifest biased language when objective reporting is substituted by subjective writing. Existing methods to detect bias mostly rely on annotated data to train machine learning models. However, low annotator agreement and comparability is a substantial drawback in available media bias corpora. To evaluate data collection options, we collect and compare labels obtained from two popular crowdsourcing platforms. Our results demonstrate the existing crowdsourcing approaches' lack of data quality, underlining the need for a trained expert framework to gather a more reliable dataset. By creating such a framework and gathering a first dataset, we are able to improve Krippendorff's α = 0.144 (crowdsourcing labels) to α = 0.419 (expert labels). We conclude that detailed annotator training increases data quality, improving the performance of existing bias detection systems. We will continue to extend our dataset in the future.