# Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression

arXiv: https://arxiv.org/abs/2108.10684 · HTML: https://mlsanity.com/paper/2108.10684/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 15 Aug 2021.

## Builds on (most cited first, 10 of 10)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia](https://mlsanity.com/paper/1909.05189/) (2019)
- [Why the World Reads Wikipedia: Beyond English Speakers](https://mlsanity.com/paper/1812.00474/) (2018)
- [The Wisdom of Polarized Crowds](https://mlsanity.com/paper/1712.06414/) (2017)
- [The Tower of Babel Meets Web 2.0: User-Generated Content and its Applications in a Multilingual Context](https://mlsanity.com/paper/1904.01689/) (2019)
- [StRE: Self Attentive Edit Quality Prediction in Wikipedia](https://mlsanity.com/paper/1906.04678/) (2019)
- [Effects of algorithmic flagging on fairness: quasi-experimental evidence from Wikipedia](https://mlsanity.com/paper/2006.03121/) (2020)
- [A Taxonomy of Knowledge Gaps for Wikimedia Projects (Second Draft)](https://mlsanity.com/paper/2008.12314/) (2020)
- [Underproduction: An Approach for Measuring Risk in Open Source Software](https://mlsanity.com/paper/2103.00352/) (2021)

## Built on by (most cited first, 1 of 1)

- [Automatic Quality Assessment of Wikipedia Articles -- A Systematic Literature Review](https://mlsanity.com/paper/2310.02235/) (2023)

## Abstract (arXiv)

Organizing complex peer production projects and advancing scientific knowledge of open collaboration each depend on the ability to measure quality. Article quality ratings on English language Wikipedia have been widely used by both Wikipedia community members and academic researchers for purposes like tracking knowledge gaps and studying how political polarization shapes collaboration. Even so, measuring quality presents many methodological challenges. The most widely used systems use labels on discrete ordinal scales when assessing quality, but such labels can be inconvenient for statistics and machine learning. Prior work handles this by assuming that different levels of quality are "evenly spaced" from one another. This assumption runs counter to intuitions about the relative degrees of effort needed to raise Wikipedia encyclopedia articles to different quality levels. Furthermore, models from prior work are fit to datasets that oversample high-quality articles. This limits their accuracy for representative samples of articles or revisions. I describe a technique extending the Wikimedia Foundations' ORES article quality model to address these limitations. My method uses weighted ordinal regression models to construct one-dimensional continuous measures of quality. While scores from my technique and from prior approaches are correlated, my approach improves accuracy for research datasets and provides evidence that the "evenly spaced" assumption is unfounded in practice on English Wikipedia. I conclude with recommendations for using quality scores in future research and include the full code, data, and models.