# A Domain-adaptive Pre-training Approach for Language Bias Detection in News

arXiv: https://arxiv.org/abs/2205.10773 · HTML: https://mlsanity.com/paper/2205.10773/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 22 May 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Multi-task learning for natural language processing in the 2020s: where are we going?](https://mlsanity.com/paper/2007.16008/) (2020)
- [MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristics](https://mlsanity.com/paper/2105.11910/) (2021)
- [Aspect-based Document Similarity for Research Papers](https://mlsanity.com/paper/2010.06395/) (2020)
- [Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection](https://mlsanity.com/paper/2103.12450/) (2021)
- [Identifying Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2103.11909/) (2021)
- [Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias](https://mlsanity.com/paper/2105.09640/) (2021)
- [Do You Think It's Biased? How To Ask For The Perception Of Media Bias](https://mlsanity.com/paper/2112.07392/) (2021)
- [Specialized Document Embeddings for Aspect-based Similarity of Research Papers](https://mlsanity.com/paper/2203.14541/) (2022)
- [Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection](https://mlsanity.com/paper/2111.07819/) (2021)
- [Towards A Reliable Ground-Truth For Biased Language Detection](https://mlsanity.com/paper/2112.07421/) (2021)
- [Identification of Biased Terms in News Articles by Comparison of Outlet-specific Word Embeddings](https://mlsanity.com/paper/2112.07384/) (2021)
- [Transforming Fake News: Robust Generalisable News Classification Using Transformers](https://mlsanity.com/paper/2109.09796/) (2021)
- [Pairwise Multi-Class Document Classification for Semantic Relations between Wikipedia Articles](https://mlsanity.com/paper/2003.09881/) (2020)
- [Incorporating Word Sense Disambiguation in Neural Language Models](https://mlsanity.com/paper/2106.07967/) (2021)

## Built on by (most cited first, 16 of 16)

- [CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain](https://mlsanity.com/paper/2212.02974/) (2022)
- [Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection](https://mlsanity.com/paper/2304.13148/) (2023)
- [The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection](https://mlsanity.com/paper/2411.11081/) (2024)
- [The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias](https://mlsanity.com/paper/2312.16148/) (2023)
- [To Bias or Not to Bias: Detecting bias in News with bias-detector](https://mlsanity.com/paper/2505.13010/) (2025)
- [MAGPIE: Multi-Task Media-Bias Analysis Generalization for Pre-Trained Identification of Expressions](https://mlsanity.com/paper/2403.07910/) (2024)
- [Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularity](https://mlsanity.com/paper/2405.16579/) (2024)
- [Data-Augmented and Retrieval-Augmented Context Enrichment in Chinese Media Bias Detection](https://mlsanity.com/paper/2311.01372/) (2023)
- [A Commonsense-Infused Language-Agnostic Learning Framework for Enhancing Prediction of Political Polarity in Multilingual News Headlines](https://mlsanity.com/paper/2212.00298/) (2022)
- [Decoding News Bias: Multi Bias Detection in News Articles](https://mlsanity.com/paper/2501.02482/) (2025)
- [LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation](https://mlsanity.com/paper/2603.12522/) (2026)
- [Reliable News or Propagandist News? A Neurosymbolic Model Using Genre, Topic, and Persuasion Techniques to Improve Robustness in Classification](https://mlsanity.com/paper/2604.01936/) (2026)
- [DocNet: Semantic Structure in Inductive Bias Detection Models](https://mlsanity.com/paper/2406.10965/) (2024)
- [Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms](https://mlsanity.com/paper/2512.23835/) (2025)
- [HierBias: Context-Conditioned Hierarchical Media Bias Detection with Multi-Task Type Classification](https://mlsanity.com/paper/2606.26100/) (2026)
- [AutoJourn: Multi-Perspective Summarisation, Bias Detection and Bias Neutralisation for LLM-Generated News in Automated Journalism](https://mlsanity.com/paper/2607.18983/) (2026)

## Abstract (arXiv)

Media bias is a multi-faceted construct influencing individual behavior and collective decision-making. Slanted news reporting is the result of one-sided and polarized writing which can occur in various forms. In this work, we focus on an important form of media bias, i.e. bias by word choice. Detecting biased word choices is a challenging task due to its linguistic complexity and the lack of representative gold-standard corpora. We present DA-RoBERTa, a new state-of-the-art transformer-based model adapted to the media bias domain which identifies sentence-level bias with an F1 score of 0.814. In addition, we also train, DA-BERT and DA-BART, two more transformer models adapted to the bias domain. Our proposed domain-adapted models outperform prior bias detection approaches on the same data.