# Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks

arXiv: https://arxiv.org/abs/2503.00124 · HTML: https://mlsanity.com/paper/2503.00124/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [User-LLM: Efficient LLM Contextualization with User Embeddings](https://mlsanity.com/paper/2402.13598/) (2024)
- [HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation](https://mlsanity.com/paper/2106.04408/) (2021)
- [Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text](https://mlsanity.com/paper/2005.11882/) (2020)
- [MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection](https://mlsanity.com/paper/2109.08113/) (2021)
- [Large Human Language Models: A Need and the Challenges](https://mlsanity.com/paper/2312.07751/) (2023)
- [A Hierarchical Self-Attentive Model for Recommending User-Generated Item Lists](https://mlsanity.com/paper/1912.13023/) (2019)
- [Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?](https://mlsanity.com/paper/2401.12492/) (2024)

## Built on by (most cited first, 3 of 3)

- [Addressing the Ecological Fallacy in Larger LMs with Human Context](https://mlsanity.com/paper/2603.05928/) (2026)
- [Who We Are, Where We Are: Mental Health at the Intersection of Person, Situation, and Large Language Models](https://mlsanity.com/paper/2603.05953/) (2026)
- [From Affect Prediction to Affect Forecasting: Evidence for Distinct Information Sources in Longitudinal Text](https://mlsanity.com/paper/2606.16687/) (2026)

## Abstract (arXiv)

Like most of NLP, models for human-centered NLP tasks -- tasks attempting to assess author-level information -- predominantly use representations derived from hidden states of Transformer-based LLMs. However, what component of the LM is used for the representation varies widely. Moreover, there is a need for Human Language Models (HuLMs) that implicitly model the author and provide a user-level hidden state. Here, we systematically evaluate different ways of representing documents and users using different LM and HuLM architectures to predict task outcomes as both dynamically changing states and averaged trait-like user-level attributes of valence, arousal, empathy, and distress. We find that representing documents as an average of the token hidden states performs the best generally. Further, while a user-level hidden state itself is rarely the best representation, we find its inclusion in the model strengthens token or document embeddings used to derive document- and user-level representations resulting in best performances.