# Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions

arXiv: https://arxiv.org/abs/2502.20897 · HTML: https://mlsanity.com/paper/2502.20897/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 79 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [DICES Dataset: Diversity in Conversational AI Evaluation for Safety](https://mlsanity.com/paper/2306.11247/) (2023)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions](https://mlsanity.com/paper/2502.16761/) (2025)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset](https://mlsanity.com/paper/2306.06826/) (2023)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs](https://mlsanity.com/paper/2311.09730/) (2023)
- [Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks](https://mlsanity.com/paper/2311.09743/) (2023)
- [Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina](https://mlsanity.com/paper/2410.19599/) (2024)

## Built on by (most cited first, 30 of 33)

- [Finetuning LLMs for Human Behavior Prediction in Social Science Experiments](https://mlsanity.com/paper/2509.05830/) (2025)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025)
- [Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation](https://mlsanity.com/paper/2603.25001/) (2026)
- [Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP](https://mlsanity.com/paper/2601.09065/) (2026)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models](https://mlsanity.com/paper/2507.13383/) (2025)
- [Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?](https://mlsanity.com/paper/2506.19467/) (2025)
- [Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives](https://mlsanity.com/paper/2508.02853/) (2025)
- [Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas](https://mlsanity.com/paper/2509.22957/) (2025)
- [German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies](https://mlsanity.com/paper/2511.21722/) (2025)
- [Improving the Distributional Alignment of LLMs using Supervision](https://mlsanity.com/paper/2507.00439/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation](https://mlsanity.com/paper/2604.07513/) (2026)
- [Synthetic Personalities: How Well Can LLMs Mimic Individual Respondents Using Socio-Economic Microdata?](https://mlsanity.com/paper/2606.04592/) (2026)
- [Opt-ICL at LeWiDi-2025: Maximizing In-Context Signal from Rater Examples via Meta-Learning](https://mlsanity.com/paper/2510.07105/) (2025)
- [Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks](https://mlsanity.com/paper/2603.21404/) (2026)
- [From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?](https://mlsanity.com/paper/2604.17968/) (2026)
- [Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation](https://mlsanity.com/paper/2605.06318/) (2026)
- [P1SCO: Social Dimensions from a Perspectivist Lens](https://mlsanity.com/paper/2605.25312/) (2026)
- [Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization](https://mlsanity.com/paper/2605.28802/) (2026)
- [Where You Go is Who You Are: Behavioral Theory-Guided LLMs for Inverse Reinforcement Learning](https://mlsanity.com/paper/2505.17249/) (2025)
- [Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions](https://mlsanity.com/paper/2509.08217/) (2025)
- [Taking a SEAT: Predicting Value Interpretations from Sentiment, Emotion, Argument, and Topic Annotations](https://mlsanity.com/paper/2510.01976/) (2025)
- [Do Androids Dream of Unseen Puppeteers? Probing for a Conspiracy Tendencies in Large Language Models](https://mlsanity.com/paper/2511.03699/) (2025)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales](https://mlsanity.com/paper/2604.21667/) (2026)
- [When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detection](https://mlsanity.com/paper/2605.27313/) (2026)
- [The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction](https://mlsanity.com/paper/2606.02911/) (2026)
- [Learning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts](https://mlsanity.com/paper/2606.22771/) (2026)

## Abstract (arXiv)

People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also been used to label data, but recent work has shown that models perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemographic knowledge. Here, we ask whether LLMs can be trained to be accurate sociodemographic models of annotator variation. Using a curated dataset of five tasks with standardized sociodemographics, we show that models do improve in sociodemographic prompting when trained but that this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemographic patterns. Across all tasks, our results suggest that models learn little meaningful connection between sociodemographics and annotation, raising doubts about the current use of LLMs for simulating sociodemographic variation and behaviour.