# Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer

arXiv: https://arxiv.org/abs/2606.11018 · HTML: https://mlsanity.com/paper/2606.11018/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 25 of 25)

- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Can Unconfident LLM Annotations Be Used for Confident Conclusions?](https://mlsanity.com/paper/2408.15204/) (2024)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [Evaluation Measures for Hierarchical Classification: a unified view and novel approaches](https://mlsanity.com/paper/1306.6802/) (2013)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts](https://mlsanity.com/paper/2407.10747/) (2024)
- [The Touch'e23-ValueEval Dataset for Identifying Human Values behind Arguments](https://mlsanity.com/paper/2301.13771/) (2023)
- [Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?](https://mlsanity.com/paper/2402.18120/) (2024)
- [Personas with Attitudes: Controlling LLMs for Diverse Data Annotation](https://mlsanity.com/paper/2410.11745/) (2024)
- [Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks](https://mlsanity.com/paper/2507.15821/) (2025)
- [Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation](https://mlsanity.com/paper/2406.12114/) (2024)
- [Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI](https://mlsanity.com/paper/2409.09467/) (2024)
- [A Unified View of Evaluation Metrics for Structured Prediction](https://mlsanity.com/paper/2310.13793/) (2023)
- [Are Chatbots Reliable Text Annotators? Sometimes](https://mlsanity.com/paper/2311.05769/) (2023)
- [Investigating Human Values in Online Communities](https://mlsanity.com/paper/2402.14177/) (2024)
- [Which Demographics do LLMs Default to During Annotation?](https://mlsanity.com/paper/2410.08820/) (2024)
- [Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks](https://mlsanity.com/paper/2505.23820/) (2025)
- [Revisiting Hierarchical Text Classification: Inference and Metrics](https://mlsanity.com/paper/2410.01305/) (2024)
- [Bias in, Bias out: Annotation Bias in Multilingual Large Language Models](https://mlsanity.com/paper/2511.14662/) (2025)
- [The Value of Nothing: Multimodal Extraction of Human Values Expressed by TikTok Influencers](https://mlsanity.com/paper/2501.11770/) (2025)
- [Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments](https://mlsanity.com/paper/2509.06704/) (2025)
- [Value Lens: Using Large Language Models to Understand Human Values](https://mlsanity.com/paper/2512.15722/) (2025)
- [Structure of Basic Human Values in Russian Social Media](https://mlsanity.com/paper/2603.18822/) (2026)

## Built on by (most cited first, 2 of 2)

- [KVpop -- Key-Value Cache Compression with Predictive Online Pruning](https://mlsanity.com/paper/2607.05061/) (2026)
- [Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study](https://mlsanity.com/paper/2607.20270/) (2026)

## Abstract (arXiv)

Measuring subjective constructs in naturally occurring social media text requires annotation procedures that are theoretically grounded, empirically validated, and transferable to an encoder model for scalable prediction. Using posts annotated according to Schwartz's theory of basic human values, we investigate how different LLMs and prompting strategies, which we call annotation regimes, operationalize the expression of values in text. Beyond standard classification metrics, we evaluate structural alignment, annotation ambiguity, error patterns, and stability across repeated runs. We find that different LLMs produce different value interpretations, and iterative prompt calibration through error analysis reduces misattributions and improves alignment with expert annotations. Error patterns are further used to derive targeted expert-verification rules for corpus annotation. We transfer soft LLM labels to an encoder model for prediction, retaining information about ambiguity in value expression. Finally, a sensitivity analysis on more than one million posts shows that regime-specific annotation differences propagate into predicted levels of value expression, whereas standardized temporal dynamics and the direction of major event responses are more robust.