Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication
Not on the map. Why?
Abstract (arXiv)
Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR: Population-Aligned Instance Replication, a post-processing method that adjusts training data to better reflect target population characteristics without collecting additional annotations. Using simulation studies on offensive language and hate speech detection with varying annotator compositions, we show that non-representative pools degrade model calibration while leaving accuracy largely unchanged. PAIR corrects these calibration problems by replicating annotations from underrepresented annotator groups to match population proportions. We conclude with recommendations for improving the representativity of training data and model performance.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 59 references that are in the map. First posted on arXiv 12 Jan 2025.
Builds on
27 of 59 references.
References · 27
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
- 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
- 2021Improving Calibration for Long-Tailed Recognitioncited by 212
- 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
- 2020Mitigating Bias in Calibration Error Estimationcited by 101
- 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
- 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
15 more
- 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
- 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
- 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 52
- 2017Using simulation studies to evaluate statistical methodscited by 49
- 2024The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labelscited by 38
- 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
- 2021Representation Matters: Assessing the Importance of Subgroup Allocations in Training Datacited by 31
- 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
- 2023Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Taskscited by 29
- 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 18
- 2023Annotation Sensitivity: Training Data Collection Methods Affect Model Performancecited by 14
- 2024Position: Insights from Survey Methodology can Improve Training Datacited by 13
- 2024How to be fair? A study of label and selection biascited by 11
- 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
- 2024Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotationcited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.