mlsanityData to 7 Oct 2026

Voices in a Crowd: Searching for Clusters of Unique Perspectives

arXiv 2407.14259 ↗First posted 19 Jul 2024cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Language models have been shown to reproduce underlying biases existing in their training data, which is the majority perspective by default. Proposed solutions aim to capture minority perspectives by either modelling annotator disagreements or grouping annotators based on shared metadata, both of which face significant challenges. We propose a framework that trains models without encoding annotator metadata, extracts latent embeddings informed by annotator behaviour, and creates clusters of similar opinions, that we refer to as voices. Resulting clusters are validated post-hoc via internal and external quantitative metrics, as well a qualitative analysis to identify the type of voice that each cluster represents. Our results demonstrate the strong generalisation capability of our framework, indicated by resulting clusters being adequately robust, while also capturing minority perspectives based on different demographic factors throughout two distinct datasets.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 38 of its 91 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

38 of 91 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  10. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  11. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  12. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
26 more
  1. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  2. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  3. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  4. 2023Aligning Language Models to User Opinionscited by 98
  5. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  6. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  7. 2018Embedding Text in Hyperbolic Spacescited by 76
  8. 2019Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claimscited by 73
  9. 2020Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!cited by 73
  10. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 52
  11. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
  12. 2020Detecting Stance in Media on Global Warmingcited by 48
  13. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  14. 2022Topic Discovery via Latent Space Clustering of Pretrained Language Model Representationscited by 34
  15. 2023Can Large Language Models Capture Dissenting Human Voices?cited by 33
  16. 2020History for Visual Dialog: Do we really need it?cited by 32
  17. 2021MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristicscited by 24
  18. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  19. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 22
  20. 2020Compositional Demographic Word Embeddingscited by 20
  21. 2019Attentive Mimicking: Better Word Embeddings by Attending to Informative Contextscited by 17
  22. 2023Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detectioncited by 10
  23. 2023iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?cited by 10
  24. 2021The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Helpcited by 6
  25. 2021Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditionscited by 6
  26. 2019Improving Document Classification with Multi-Sense Embeddingscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Value Profiles for Encoding Human Variationcited by 29
  2. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  3. 2025NUTMEG: Separating Signal From Noise in Annotator Disagreementcited by 5
  4. 2025Explaining Matters: Leveraging Definitions and Semantic Expansion for Sexism Detectioncited by 2
  5. 2025Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replicationcited by 1
  6. 2025Belief-Calibrated Multi-Agent Consensus Seeking for Complex NLP Taskscited by 1
  7. 2026Labels have Human Values: Value Calibration of Subjective Taskscited by 1
  8. 2026Structuring the Space of Perspectivescited by 1
  9. 2026Evaluating Pluralism in LLMs through Latent Perspectivescited by 0