# To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction

arXiv: https://arxiv.org/abs/2408.02257 · HTML: https://mlsanity.com/paper/2408.02257/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023)
- [Weakly- and Semi-supervised Evidence Extraction](https://mlsanity.com/paper/2011.01459/) (2020)

## Built on by (most cited first, 2 of 2)

- [Training and Evaluating with Human Label Variation: An Empirical Study](https://mlsanity.com/paper/2502.01891/) (2025)
- [Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification](https://mlsanity.com/paper/2605.17691/) (2026)

## Abstract (arXiv)

This paper explores the task of automatic prediction of text spans in a legal problem description that support a legal area label. We use a corpus of problem descriptions written by laypeople in English that is annotated by practising lawyers. Inherent subjectivity exists in our task because legal area categorisation is a complex task, and lawyers often have different views on a problem, especially in the face of legally-imprecise descriptions of issues. Experiments show that training on majority-voted spans outperforms training on disaggregated ones.