LPI-RIT at LeWiDi-2025: Improving Distributional Predictions via Metadata and Loss Reweighting with DisCo
Not on the map. Why?
Abstract (arXiv)
The Learning With Disagreements (LeWiDi) 2025 shared task aims to model annotator disagreement through soft label distribution prediction and perspectivist evaluation, which focuses on modeling individual annotators. We adapt DisCo (Distribution from Context), a neural architecture that jointly models item-level and annotator-level label distributions, and present detailed analysis and improvements. In this paper, we extend DisCo by introducing annotator metadata embeddings, enhancing input representations, and multi-objective training losses to capture disagreement patterns better. Through extensive experiments, we demonstrate substantial improvements in both soft and perspectivist evaluation metrics across three datasets. We also conduct in-depth calibration and error analyses that reveal when and why disagreement-aware modeling improves. Our findings show that disagreement can be better captured by conditioning on annotator demographics and by optimizing directly for distributional metrics, yielding consistent improvements across datasets.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 8 of its 30 references that are in the map. First posted on arXiv 11 Aug 2025.
Builds on
8 of 30 references.
References · 8
- 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 130
- 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
- 2014Label Distribution Learningcited by 124
- 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 111
- 2016Deep Label Distribution Learning with Label Ambiguitycited by 104
- 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
- 2023Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safetycited by 8
- 2024Rater Cohesion and Quality from a Vicarious Perspectivecited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.