Modeling Subjectivity in Cognitive Appraisal with Language Models
Not on the map. Why?
Abstract (arXiv)
As the utilization of language models in interdisciplinary, human-centered studies grow, expectations of their capabilities continue to evolve. Beyond excelling at conventional tasks, models are now expected to perform well on user-centric measurements involving confidence and human (dis)agreement-factors that reflect subjective preferences. While modeling subjectivity plays an essential role in cognitive science and has been extensively studied, its investigation at the intersection with NLP remains under-explored. In light of this gap, we explore how language models can quantify subjectivity in cognitive appraisal by conducting comprehensive experiments and analyses with both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative results demonstrate that personality traits and demographic information are critical for measuring subjectivity, yet existing post-hoc calibration methods often fail to achieve satisfactory performance. Furthermore, our in-depth analysis provides valuable insights to guide future research at the intersection of NLP and cognitive science.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 65 references that are in the map. First posted on arXiv 14 Mar 2025.
Builds on
24 of 65 references.
References · 24
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2024Qwen2 Technical Reportcited by 2,120
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
- 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
- 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
- 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 81
- 2023Cognitive Reframing of Negative Thoughts through Human-Language Model Interactioncited by 77
12 more
- 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 72
- 2022Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Predictioncited by 46
- 2023Human Uncertainty in Concept-Based AI Systemscited by 39
- 2024GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theorycited by 37
- 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
- 2020Appraisal Theories for Emotion Classification in Textcited by 28
- 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
- 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
- 2024Large Language Models are Capable of Offering Cognitive Reappraisal, if Guidedcited by 13
- 2024Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoptioncited by 9
- 2025EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity Statescited by 6
- 2024The Interaction Layer: An Exploration for Co-Designing User-LLM Interactions in Parental Wellbeing Support Systemscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.