# Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?

arXiv: https://arxiv.org/abs/2401.12492 · HTML: https://mlsanity.com/paper/2401.12492/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 14 of 14)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Personalized Transformer for Explainable Recommendation](https://mlsanity.com/paper/2105.11601/) (2021)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [Personalized Language Model for Query Auto-Completion](https://mlsanity.com/paper/1804.09661/) (2018)
- [EmpathBERT: A BERT-based Framework for Demographic-aware Empathy Prediction](https://mlsanity.com/paper/2102.00272/) (2021)
- [MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection](https://mlsanity.com/paper/2109.08113/) (2021)
- [Large Human Language Models: A Need and the Challenges](https://mlsanity.com/paper/2312.07751/) (2023)
- [Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers](https://mlsanity.com/paper/2210.07362/) (2022)
- [Freshman or Fresher? Quantifying the Geographic Variation of Internet Language](https://mlsanity.com/paper/1510.06786/) (2015)
- [Learning Dynamic Author Representations with Temporal Language Models](https://mlsanity.com/paper/1909.04985/) (2019)

## Built on by (most cited first, 4 of 4)

- [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025)
- [Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks](https://mlsanity.com/paper/2503.00124/) (2025)
- [Addressing the Ecological Fallacy in Larger LMs with Human Context](https://mlsanity.com/paper/2603.05928/) (2026)
- [Who We Are, Where We Are: Mental Health at the Intersection of Person, Situation, and Large Language Models](https://mlsanity.com/paper/2603.05953/) (2026)

## Abstract (arXiv)

Pre-trained language models consider the context of neighboring words and documents but lack any author context of the human generating the text. However, language depends on the author's states, traits, social, situational, and environmental attributes, collectively referred to as human context (Soni et al., 2024). Human-centered natural language processing requires incorporating human context into language models. Currently, two methods exist: pre-training with 1) group-wise attributes (e.g., over-45-year-olds) or 2) individual traits. Group attributes are simple but coarse -- not all 45-year-olds write the same way -- while individual traits allow for more personalized representations, but require more complex modeling and data. It is unclear which approach benefits what tasks. We compare pre-training models with human context via 1) group attributes, 2) individual users, and 3) a combined approach on five user- and document-level tasks. Our results show that there is no best approach, but that human-centered language modeling holds avenues for different methods.