mlsanityData to 7 Oct 2026

Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP

arXiv 2603.28351 ↗First posted 30 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Subjective judgments are part of several NLP datasets and recent work is increasingly prioritizing models whose outputs reflect this diversity of perspectives. Such responses allow us to shed light on minority voices, which are frequently marginalized or obscured by dominant perspectives. It remains a question whether our evaluation practices align with these models' objectives. This position paper proposes seven evaluation desiderata for subjectivity-sensitive models, rooted in how subjectivity is represented in NLP data and models. The desiderata are constructed in a top-down approach, keeping in mind the user-centric impact of such models. We scan the experimental setup of 60 papers and show that various aspects of subjectivity are still understudied: the distinction between ambiguous and polyphonic input, whether subjectivity is effectively expressed to the user, and a lack of interplay between different desiderata, amongst other gaps.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 82 of its 142 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

82 of 142 references.

References · 82

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  5. 2023Whose Opinions Do Language Models Reflect?cited by 759
  6. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  7. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  8. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  9. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  10. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  11. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  12. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
38 more
  1. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  2. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  3. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  4. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  5. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  6. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  7. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  8. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  9. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  10. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  11. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  12. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  13. 2017Deep learning from crowdscited by 105
  14. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 100
  15. 2024Can LLM be a Personalized Judge?cited by 99
  16. 2023Aligning Language Models to User Opinionscited by 98
  17. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
  18. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  19. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  20. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  21. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 89
  22. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
  23. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  24. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  25. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  26. 2022Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotationcited by 67
  27. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  28. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  29. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  30. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  31. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 51
  32. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  33. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 44
  34. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  35. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  36. 2023Learning to Rewrite Prompts for Personalized Text Generationcited by 40
  37. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  38. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34

50 most cited of 82 All 82 (JSON)

Built on by

Not cited yet.