# Language-Agnostic Modeling of Source Reliability on Wikipedia

arXiv: https://arxiv.org/abs/2410.18803 · HTML: https://mlsanity.com/paper/2410.18803/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 78 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability](https://mlsanity.com/paper/1902.11116/) (2019)
- [Studying Fake News via Network Analysis: Detection and Mitigation](https://mlsanity.com/paper/1804.10233/) (2018)
- [WikiCheck: An end-to-end open source Automatic Fact-Checking API based on Wikipedia](https://mlsanity.com/paper/2109.00835/) (2021)
- [Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection](https://mlsanity.com/paper/2309.16424/) (2023)
- [Multilinguals and Wikipedia Editing](https://mlsanity.com/paper/1312.0976/) (2013)
- [MediaRank: Computational Ranking of Online News Sources](https://mlsanity.com/paper/1903.07581/) (2019)
- [Language-agnostic Topic Classification for Wikipedia](https://mlsanity.com/paper/2103.00068/) (2021)
- [Wiki-Reliability: A Large Scale Dataset for Content Reliability on Wikipedia](https://mlsanity.com/paper/2105.04117/) (2021)
- [Identifying Cost-effective Debunkers for Multi-stage Fake News Mitigation Campaigns](https://mlsanity.com/paper/2203.16854/) (2022)
- [Language-Agnostic Modeling of Wikipedia Articles for Content Quality Assessment across Languages](https://mlsanity.com/paper/2404.09764/) (2024)
- [Wikimedia data for AI: a review of Wikimedia datasets for NLP tasks and AI-assisted editing](https://mlsanity.com/paper/2410.08918/) (2024)
- [Crosslingual Topic Modeling with WikiPDA](https://mlsanity.com/paper/2009.11207/) (2020)
- [A Multilingual Entity Linking System for Wikipedia with a Machine-in-the-Loop Approach](https://mlsanity.com/paper/2105.15110/) (2021)
- [Longitudinal Assessment of Reference Quality on Wikipedia](https://mlsanity.com/paper/2303.05227/) (2023)
- [Fair multilingual vandalism detection system for Wikipedia](https://mlsanity.com/paper/2306.01650/) (2023)
- [Governance Capture in a Self-Governing Community: A Qualitative Comparison of the Serbo-Croatian Wikipedias](https://mlsanity.com/paper/2311.03616/) (2023)
- [Polarization and reliability of news sources in Wikipedia](https://mlsanity.com/paper/2210.16065/) (2022)
- [Orphan Articles: The Dark Matter of Wikipedia](https://mlsanity.com/paper/2306.03940/) (2023)
- [Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia](https://mlsanity.com/paper/2410.04254/) (2024)
- [Laboratories of Oligarchy? How the Iron Law Extends to Peer Production](https://mlsanity.com/paper/1407.0323/) (2014)
- [A Comparative Study of Reference Reliability in Multiple Language Editions of Wikipedia](https://mlsanity.com/paper/2309.00196/) (2023)
- [Societal Controversies in Wikipedia Articles](https://mlsanity.com/paper/1904.08721/) (2019)
- [A preliminary approach to knowledge integrity risk assessment in Wikipedia projects](https://mlsanity.com/paper/2106.15940/) (2021)
- [A Commonsense-Infused Language-Agnostic Learning Framework for Enhancing Prediction of Political Polarity in Multilingual News Headlines](https://mlsanity.com/paper/2212.00298/) (2022)
- [The Dynamics of (Not) Unfollowing Misinformation Spreaders](https://mlsanity.com/paper/2401.13480/) (2024)

## Abstract (arXiv)

Over the last few years, verifying the credibility of information sources has become a fundamental need to combat disinformation. Here, we present a language-agnostic model designed to assess the reliability of web domains as sources in references across multiple language editions of Wikipedia. Utilizing editing activity data, the model evaluates domain reliability within different articles of varying controversiality, such as Climate Change, COVID-19, History, Media, and Biology topics. Crafting features that express domain usage across articles, the model effectively predicts domain reliability, achieving an F1 Macro score of approximately 0.80 for English and other high-resource languages. For mid-resource languages, we achieve 0.65, while the performance of low-resource languages varies. In all cases, the time the domain remains present in the articles (which we dub as permanence) is one of the most predictive features. We highlight the challenge of maintaining consistent model performance across languages of varying resource levels and demonstrate that adapting models from higher-resource languages can improve performance. We believe these findings can assist Wikipedia editors in their ongoing efforts to verify citations and may offer useful insights for other user-generated content communities.