# It's All Relative: Interpretable Models for Scoring Bias in Documents

arXiv: https://arxiv.org/abs/2307.08139 · HTML: https://mlsanity.com/paper/2307.08139/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [Text-Based Ideal Points](https://mlsanity.com/paper/2005.04232/) (2020)
- [Wiki-Reliability: A Large Scale Dataset for Content Reliability on Wikipedia](https://mlsanity.com/paper/2105.04117/) (2021)
- [Mind Your POV: Convergence of Articles and Editors Towards Wikipedia's Neutrality Norm](https://mlsanity.com/paper/1809.06951/) (2018)
- [Can Who-Edits-What Predict Edit Survival?](https://mlsanity.com/paper/1801.04159/) (2018)

## Abstract (arXiv)

We propose an interpretable model to score the bias present in web documents, based only on their textual content. Our model incorporates assumptions reminiscent of the Bradley-Terry axioms and is trained on pairs of revisions of the same Wikipedia article, where one version is more biased than the other. While prior approaches based on absolute bias classification have struggled to obtain a high accuracy for the task, we are able to develop a useful model for scoring bias by learning to perform pairwise comparisons of bias accurately. We show that we can interpret the parameters of the trained model to discover the words most indicative of bias. We also apply our model in three different settings - studying the temporal evolution of bias in Wikipedia articles, comparing news sources based on bias, and scoring bias in law amendments. In each case, we demonstrate that the outputs of the model can be explained and validated, even for the two domains that are outside the training-data domain. We also use the model to compare the general level of bias between domains, where we see that legal texts are the least biased and news media are the most biased, with Wikipedia articles in between. Given its high performance, simplicity, interpretability, and wide applicability, we hope the model will be useful for a large community, including Wikipedia and news editors, political and social scientists, and the general public.