# Modeling Subjectivity in Cognitive Appraisal with Language Models

arXiv: https://arxiv.org/abs/2503.11381 · HTML: https://mlsanity.com/paper/2503.11381/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 65 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [LUQ: Long-text Uncertainty Quantification for LLMs](https://mlsanity.com/paper/2403.20279/) (2024)
- [Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction](https://mlsanity.com/paper/2305.02466/) (2023)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021)
- [Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction](https://mlsanity.com/paper/2206.05238/) (2022)
- [Human Uncertainty in Concept-Based AI Systems](https://mlsanity.com/paper/2303.12872/) (2023)
- [GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory](https://mlsanity.com/paper/2406.11149/) (2024)
- [The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis](https://mlsanity.com/paper/2311.00237/) (2023)
- [Appraisal Theories for Emotion Classification in Text](https://mlsanity.com/paper/2003.14155/) (2020)
- [Aligning Language Models with Human Preferences via a Bayesian Approach](https://mlsanity.com/paper/2310.05782/) (2023)
- [Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas](https://mlsanity.com/paper/2406.14462/) (2024)
- [Large Language Models are Capable of Offering Cognitive Reappraisal, if Guided](https://mlsanity.com/paper/2404.01288/) (2024)
- [Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoption](https://mlsanity.com/paper/2409.03843/) (2024)
- [EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity States](https://mlsanity.com/paper/2503.03340/) (2025)
- [The Interaction Layer: An Exploration for Co-Designing User-LLM Interactions in Parental Wellbeing Support Systems](https://mlsanity.com/paper/2411.01228/) (2024)

## Built on by (most cited first, 3 of 3)

- [MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs](https://mlsanity.com/paper/2505.24858/) (2025)
- [CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning](https://mlsanity.com/paper/2605.17176/) (2026)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)

## Abstract (arXiv)

As the utilization of language models in interdisciplinary, human-centered studies grow, expectations of their capabilities continue to evolve. Beyond excelling at conventional tasks, models are now expected to perform well on user-centric measurements involving confidence and human (dis)agreement-factors that reflect subjective preferences. While modeling subjectivity plays an essential role in cognitive science and has been extensively studied, its investigation at the intersection with NLP remains under-explored. In light of this gap, we explore how language models can quantify subjectivity in cognitive appraisal by conducting comprehensive experiments and analyses with both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative results demonstrate that personality traits and demographic information are critical for measuring subjectivity, yet existing post-hoc calibration methods often fail to achieve satisfactory performance. Furthermore, our in-depth analysis provides valuable insights to guide future research at the intersection of NLP and cognitive science.