# Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection

arXiv: https://arxiv.org/abs/2304.13148 · HTML: https://mlsanity.com/paper/2304.13148/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 29 of 29)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://mlsanity.com/paper/2008.06460/) (2020)
- [DynaSent: A Dynamic Benchmark for Sentiment Analysis](https://mlsanity.com/paper/2012.15349/) (2020)
- [Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Experts](https://mlsanity.com/paper/2209.14557/) (2022)
- [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://mlsanity.com/paper/2205.00619/) (2022)
- ["Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples](https://mlsanity.com/paper/2004.12764/) (2020)
- [A Systematic Media Frame Analysis of 1.5 Million New York Times Articles from 2000 to 2017](https://mlsanity.com/paper/2005.01803/) (2020)
- [Neural Based Statement Classification for Biased Language](https://mlsanity.com/paper/1811.05740/) (2018)
- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [Female Librarians and Male Computer Programmers? Gender Bias in Occupational Images on Digital Media Platforms](https://mlsanity.com/paper/1912.05474/) (2019)
- [Measuring and Characterizing Hate Speech on News Websites](https://mlsanity.com/paper/2005.07926/) (2020)
- [Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles](https://mlsanity.com/paper/2211.03491/) (2022)
- [Automatic Detection of Sexist Statements Commonly Used at the Workplace](https://mlsanity.com/paper/2007.04181/) (2020)
- [Towards A Reliable Ground-Truth For Biased Language Detection](https://mlsanity.com/paper/2112.07421/) (2021)
- [Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions](https://mlsanity.com/paper/2101.11956/) (2021)
- [Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity](https://mlsanity.com/paper/2104.08829/) (2021)
- [Mitigating Media Bias through Neutral Article Generation](https://mlsanity.com/paper/2104.00336/) (2021)
- [An Interdisciplinary Approach for the Automated Detection and Visualization of Media Bias in News Articles](https://mlsanity.com/paper/2112.13352/) (2021)
- [TASSY -- A Text Annotation Survey System](https://mlsanity.com/paper/2112.07391/) (2021)

## Built on by (most cited first, 18 of 18)

- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators](https://mlsanity.com/paper/2402.00345/) (2024)
- [The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection](https://mlsanity.com/paper/2411.11081/) (2024)
- [MAGPIE: Multi-Task Media-Bias Analysis Generalization for Pre-Trained Identification of Expressions](https://mlsanity.com/paper/2403.07910/) (2024)
- [BEADs: Bias Evaluation Across Domains](https://mlsanity.com/paper/2406.04220/) (2024)
- [Experiments in News Bias Detection with Pre-Trained Neural Transformers](https://mlsanity.com/paper/2406.09938/) (2024)
- [ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models](https://mlsanity.com/paper/2403.20158/) (2024)
- [Mapping the Media Landscape: Predicting Factual Reporting and Political Bias Through Web Interactions](https://mlsanity.com/paper/2410.17655/) (2024)
- [AdvSumm: Adversarial Training for Bias Mitigation in Text Summarization](https://mlsanity.com/paper/2506.06273/) (2025)
- [BiasScanner: Automatic Detection and Classification of News Bias to Strengthen Democracy](https://mlsanity.com/paper/2407.10829/) (2024)
- [Navigating Nuance: In Quest for Political Truth](https://mlsanity.com/paper/2501.00782/) (2025)
- [DocNet: Semantic Structure in Inductive Bias Detection Models](https://mlsanity.com/paper/2406.10965/) (2024)
- [Intertwined Biases Across Social Media Spheres: Unpacking Correlations in Media Bias Dimensions](https://mlsanity.com/paper/2408.15406/) (2024)
- [On Adversarial Robustness of Language Models in Transfer Learning](https://mlsanity.com/paper/2501.00066/) (2024)
- [HierBias: Context-Conditioned Hierarchical Media Bias Detection with Multi-Task Type Classification](https://mlsanity.com/paper/2606.26100/) (2026)
- [Can temporal article-level credibility signals improve domain-level credibility prediction?](https://mlsanity.com/paper/2607.04560/) (2026)
- [MABPD: Multi-Agent Bias Probing & Detection via Structured Argument Debate](https://mlsanity.com/paper/2609.04841/) (2026)
- [Does Machine Translation Retain Media Bias?](https://mlsanity.com/paper/2610.19945/) (2026)

## Abstract (arXiv)

Although media bias detection is a complex multi-task problem, there is, to date, no unified benchmark grouping these evaluation tasks. We introduce the Media Bias Identification Benchmark (MBIB), a comprehensive benchmark that groups different types of media bias (e.g., linguistic, cognitive, political) under a common framework to test how prospective detection techniques generalize. After reviewing 115 datasets, we select nine tasks and carefully propose 22 associated datasets for evaluating media bias detection techniques. We evaluate MBIB using state-of-the-art Transformer techniques (e.g., T5, BART). Our results suggest that while hate speech, racial bias, and gender bias are easier to detect, models struggle to handle certain bias types, e.g., cognitive and political bias. However, our results show that no single technique can outperform all the others significantly. We also find an uneven distribution of research interest and resource allocation to the individual tasks in media bias. A unified benchmark encourages the development of more robust systems and shifts the current paradigm in media bias detection evaluation towards solutions that tackle not one but multiple media bias types simultaneously.