# Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

arXiv: https://arxiv.org/abs/2509.06704 · HTML: https://mlsanity.com/paper/2509.06704/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 63 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [MoralStrength: Exploiting a Moral Lexicon and Embedding Similarity for Moral Foundations Prediction](https://mlsanity.com/paper/1904.08314/) (2019)
- [The Touch'e23-ValueEval Dataset for Identifying Human Values behind Arguments](https://mlsanity.com/paper/2301.13771/) (2023)
- [The Moral Debater: A Study on the Computational Generation of Morally Framed Arguments](https://mlsanity.com/paper/2203.14563/) (2022)
- [Annotator-Centric Active Learning for Subjective NLP Tasks](https://mlsanity.com/paper/2404.15720/) (2024)
- [ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation](https://mlsanity.com/paper/2310.14979/) (2023)
- [Learning to Adapt Domain Shifts of Moral Values via Instance Weighting](https://mlsanity.com/paper/2204.07603/) (2022)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [Value Preferences Estimation and Disambiguation in Hybrid Participatory Systems](https://mlsanity.com/paper/2402.16751/) (2024)
- [EMONA: Event-level Moral Opinions in News Articles](https://mlsanity.com/paper/2404.01715/) (2024)

## Built on by (most cited first, 3 of 3)

- [Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer](https://mlsanity.com/paper/2606.11018/) (2026)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations](https://mlsanity.com/paper/2610.19817/) (2026)

## Abstract (arXiv)

Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.