# WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification

arXiv: https://arxiv.org/abs/2605.26070 · HTML: https://mlsanity.com/paper/2605.26070/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 10 of 10)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis](https://mlsanity.com/paper/2308.16705/) (2023)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks](https://mlsanity.com/paper/2507.15821/) (2025)
- [The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?](https://mlsanity.com/paper/2410.04699/) (2024)
- [NEREL: A Russian Dataset with Nested Named Entities, Relations and Events](https://mlsanity.com/paper/2108.13112/) (2021)
- [Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems](https://mlsanity.com/paper/2305.05917/) (2023)
- [Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains](https://mlsanity.com/paper/2311.08704/) (2023)
- [Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications](https://mlsanity.com/paper/2403.06570/) (2024)

## Abstract (arXiv)

Annotating speaker attributes from text is inherently ambiguous, particularly in multilingual settings where demographic and social cues are implicit and culturally variable. We propose a human-large language model (LLM) collaborative re-annotation framework for stabilizing multilingual speaker-attribute labels under practical resource constraints. Starting from a noisy corpus, we use LLMs to surface recurring annotation rationales through iterative interaction with experts, and apply disagreement-focused sampling for targeted re-annotation. Using this framework, we construct WhoSaidIt, a multilingual dataset covering nine speaker-attribute labels. We quantify divergence between original and revised annotations, benchmark recent LLMs, and analyze the effect of explicit rationales on model behavior. Our results reveal substantial cross-lingual differences in annotation decisions and demonstrate both the strengths and limitations of LLMs in speaker-attribute classification.