# The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection

arXiv: https://arxiv.org/abs/2411.11081 · HTML: https://mlsanity.com/paper/2411.11081/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 25 of 25)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development](https://mlsanity.com/paper/2108.04308/) (2021)
- [In Plain Sight: Media Bias Through the Lens of Factual Reporting](https://mlsanity.com/paper/1909.02670/) (2019)
- [Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Experts](https://mlsanity.com/paper/2209.14557/) (2022)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristics](https://mlsanity.com/paper/2105.11910/) (2021)
- [Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP](https://mlsanity.com/paper/2305.01633/) (2023)
- [Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection](https://mlsanity.com/paper/2304.13148/) (2023)
- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators](https://mlsanity.com/paper/2402.00345/) (2024)
- [Do You Think It's Biased? How To Ask For The Perception Of Media Bias](https://mlsanity.com/paper/2112.07392/) (2021)
- [The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias](https://mlsanity.com/paper/2312.16148/) (2023)
- [Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles](https://mlsanity.com/paper/2211.03491/) (2022)
- [Towards A Reliable Ground-Truth For Biased Language Detection](https://mlsanity.com/paper/2112.07421/) (2021)
- [MAGPIE: Multi-Task Media-Bias Analysis Generalization for Pre-Trained Identification of Expressions](https://mlsanity.com/paper/2403.07910/) (2024)
- [Sentence-level Media Bias Analysis with Event Relation Graph](https://mlsanity.com/paper/2404.01722/) (2024)

## Built on by (most cited first, 15 of 15)

- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs](https://mlsanity.com/paper/2601.17058/) (2026)
- [FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation](https://mlsanity.com/paper/2602.23636/) (2026)
- [Leveraging Large Language Models for Automated Definition Extraction with TaxoMatic A Case Study on Media Bias](https://mlsanity.com/paper/2504.00343/) (2025)
- [Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models](https://mlsanity.com/paper/2506.08593/) (2025)
- [Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale](https://mlsanity.com/paper/2508.05938/) (2025)
- [Are Large Language Models the future crowd workers of Linguistics?](https://mlsanity.com/paper/2502.10266/) (2025)
- [Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets](https://mlsanity.com/paper/2503.16674/) (2025)
- [PartisanLens: A Multilingual Dataset of Hyperpartisan and Conspiratorial Immigration Narratives in European Media](https://mlsanity.com/paper/2601.03860/) (2026)
- [C-Mining: Unsupervised Discovery of Seeds for Cultural Data Synthesis via Geometric Misalignment](https://mlsanity.com/paper/2604.15675/) (2026)
- [HierBias: Context-Conditioned Hierarchical Media Bias Detection with Multi-Task Type Classification](https://mlsanity.com/paper/2606.26100/) (2026)
- [Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark](https://mlsanity.com/paper/2608.23095/) (2026)
- [CALICO: A Human-Centered, Codebook-Aligned System for Annotation](https://mlsanity.com/paper/2609.14726/) (2026)
- [Synthetic Data Characterization via Training Dynamics](https://mlsanity.com/paper/2609.39447/) (2026)
- [Collective Bias Mitigation via Model Routing and Collaboration](https://mlsanity.com/paper/2610.03240/) (2026)

## Abstract (arXiv)

High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.