# LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task

arXiv: https://arxiv.org/abs/2510.08460 · HTML: https://mlsanity.com/paper/2510.08460/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 56 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021)
- [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023)
- [What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability](https://mlsanity.com/paper/2305.11707/) (2023)
- [ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI](https://mlsanity.com/paper/2109.09483/) (2021)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021)
- [Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability](https://mlsanity.com/paper/2510.06084/) (2025)
- [A Crowdsourced Frame Disambiguation Corpus with Ambiguity](https://mlsanity.com/paper/1904.06101/) (2019)
- [Lexicosyntactic Inference in Neural Models](https://mlsanity.com/paper/1808.06232/) (2018)
- [LPI-RIT at LeWiDi-2025: Improving Distributional Predictions via Metadata and Loss Reweighting with DisCo](https://mlsanity.com/paper/2508.08163/) (2025)
- [Opt-ICL at LeWiDi-2025: Maximizing In-Context Signal from Rater Examples via Meta-Learning](https://mlsanity.com/paper/2510.07105/) (2025)

## Built on by (most cited first, 9 of 9)

- [Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis](https://mlsanity.com/paper/2605.10415/) (2026)
- [Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization](https://mlsanity.com/paper/2605.28802/) (2026)
- [Bayesian Spectral Emotion Transition Discovery from Multi-Annotator Disagreement](https://mlsanity.com/paper/2606.01906/) (2026)
- [Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework](https://mlsanity.com/paper/2606.21069/) (2026)
- [The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction](https://mlsanity.com/paper/2606.02911/) (2026)
- [Multimodal Sexism Identification and Characterization using Large Language Models and Gradient Boosting](https://mlsanity.com/paper/2606.05997/) (2026)
- [Ensemble Diversity Optimization for Subjective Supervision](https://mlsanity.com/paper/2607.08493/) (2026)
- [Decomposing LLM-Judge Uncertainty to Target Expert Labels](https://mlsanity.com/paper/2609.06444/) (2026)
- [Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages](https://mlsanity.com/paper/2610.18588/) (2026)

## Abstract (arXiv)

Many researchers have reached the conclusion that AI models should be trained to be aware of the possibility of variation and disagreement in human judgments, and evaluated as per their ability to recognize such variation. The LEWIDI series of shared tasks on Learning With Disagreements was established to promote this approach to training and evaluating AI models, by making suitable datasets more accessible and by developing evaluation methods. The third edition of the task builds on this goal by extending the LEWIDI benchmark to four datasets spanning paraphrase identification, irony detection, sarcasm detection, and natural language inference, with labeling schemes that include not only categorical judgments as in previous editions, but ordinal judgments as well. Another novelty is that we adopt two complementary paradigms to evaluate disagreement-aware systems: the soft-label approach, in which models predict population-level distributions of judgments, and the perspectivist approach, in which models predict the interpretations of individual annotators. Crucially, we moved beyond standard metrics such as cross-entropy, and tested new evaluation metrics for the two paradigms. The task attracted diverse participation, and the results provide insights into the strengths and limitations of methods to modeling variation. Together, these contributions strengthen LEWIDI as a framework and provide new resources, benchmarks, and findings to support the development of disagreement-aware technologies.