# Evaluating Fairness in Argument Retrieval

arXiv: https://arxiv.org/abs/2108.10442 · HTML: https://mlsanity.com/paper/2108.10442/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2021.

## Built on by (most cited first, 3 of 3)

- [Properties of Group Fairness Metrics for Rankings](https://mlsanity.com/paper/2212.14351/) (2022)
- [Overview of PerpectiveArg2024: The First Shared Task on Perspective Argument Retrieval](https://mlsanity.com/paper/2407.19670/) (2024)
- [Navigating the Thin Line: Examining User Behavior in Search to Detect Engagement and Backfire Effects](https://mlsanity.com/paper/2401.11201/) (2024)

## Abstract (arXiv)

Existing commercial search engines often struggle to represent different perspectives of a search query. Argument retrieval systems address this limitation of search engines and provide both positive (PRO) and negative (CON) perspectives about a user's information need on a controversial topic (e.g., climate change). The effectiveness of such argument retrieval systems is typically evaluated based on topical relevance and argument quality, without taking into account the often differing number of documents shown for the argument stances (PRO or CON). Therefore, systems may retrieve relevant passages, but with a biased exposure of arguments. In this work, we analyze a range of non-stochastic fairness-aware ranking and diversity metrics to evaluate the extent to which argument stances are fairly exposed in argument retrieval systems. Using the official runs of the argument retrieval task Touch'e at CLEF 2020, as well as synthetic data to control the amount and order of argument stances in the rankings, we show that systems with the best effectiveness in terms of topical relevance are not necessarily the most fair or the most diverse in terms of argument stance. The relationships we found between (un)fairness and diversity metrics shed light on how to evaluate group fairness -- in addition to topical relevance -- in argument retrieval settings.