# Mitigating Voter Attribute Bias for Fair Opinion Aggregation

arXiv: https://arxiv.org/abs/2307.10749 · HTML: https://mlsanity.com/paper/2307.10749/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 48 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 11 of 11)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [Toward Annotator Group Bias in Crowdsourcing](https://mlsanity.com/paper/2110.08038/) (2021)
- [Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty](https://mlsanity.com/paper/2108.01799/) (2021)
- [Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations](https://mlsanity.com/paper/1902.08918/) (2019)

## Built on by (most cited first, 2 of 2)

- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [REQUAL-LM: Reliability and Equity through Aggregation in Large Language Models](https://mlsanity.com/paper/2404.11782/) (2024)

## Abstract (arXiv)

The aggregation of multiple opinions plays a crucial role in decision-making, such as in hiring and loan review, and in labeling data for supervised learning. Although majority voting and existing opinion aggregation models are effective for simple tasks, they are inappropriate for tasks without objectively true labels in which disagreements may occur. In particular, when voter attributes such as gender or race introduce bias into opinions, the aggregation results may vary depending on the composition of voter attributes. A balanced group of voters is desirable for fair aggregation results but may be difficult to prepare. In this study, we consider methods to achieve fair opinion aggregation based on voter attributes and evaluate the fairness of the aggregated results. To this end, we consider an approach that combines opinion aggregation models such as majority voting and the Dawid and Skene model (D&S model) with fairness options such as sample weighting. To evaluate the fairness of opinion aggregation, probabilistic soft labels are preferred over discrete class labels. First, we address the problem of soft label estimation without considering voter attributes and identify some issues with the D&S model. To address these limitations, we propose a new Soft D&S model with improved accuracy in estimating soft labels. Moreover, we evaluated the fairness of an opinion aggregation model, including Soft D&S, in combination with different fairness options using synthetic and semi-synthetic data. The experimental results suggest that the combination of Soft D&S and data splitting as a fairness option is effective for dense data, whereas weighted majority voting is effective for sparse data. These findings should prove particularly valuable in supporting decision-making by human and machine-learning models with balanced opinion aggregation.