mlsanityData to 14 Oct 2026

Cognitive models can reveal interpretable value trade-offs in language models

arXiv 2506.20666 ↗First posted 25 Jun 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Value trade-offs are an integral part of human decision-making and language use, however, current tools for interpreting such dynamic and multi-faceted notions of values in language models are limited. In cognitive science, so-called "cognitive models" provide formal accounts of such trade-offs in humans, by modeling the weighting of a speaker's competing utility functions in choosing an action or utterance. Here, we show that a leading cognitive model of polite speech can be used to systematically evaluate alignment-relevant trade-offs in language models via two encompassing settings: degrees of reasoning "effort" and system prompt manipulations in closed-source frontier models, and RL post-training dynamics of open-source models. Our results show that LLMs' behavioral profiles under the cognitive model a) shift predictably when they are prompted to prioritize certain goals, b) are amplified by a small reasoning budget, and c) can be used to diagnose other social behaviors such as sycophancy. Our findings from LLMs' post-training dynamics reveal large shifts in values early on in training and persistent effects of the choice of base model and pretraining data, compared to feedback dataset or alignment method. Our framework offers a flexible tool for probing behavioral profiles across diverse model types and gaining insights for shaping training regimes that better control trade-offs between values during model development.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 50 of its 93 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

50 of 93 references.

References · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
38 more
  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  6. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  9. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  11. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  12. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
  13. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
  14. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  15. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  16. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  17. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 101
  18. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
  19. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  20. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  21. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  22. 2024Beyond Preferences in AI Alignmentcited by 49
  23. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  24. 2023Diminished Diversity-of-Thought in a Standard Large Language Modelcited by 45
  25. 2023Prompting is not a substitute for probability measurements in large language modelscited by 44
  26. 2025TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Modelscited by 33
  27. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
  28. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  29. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  30. 2024In-context learning agents are asymmetric belief updaterscited by 14
  31. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 11
  32. 2023Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspectivecited by 10
  33. 2025Comparing human and LLM politeness strategies in free productioncited by 9
  34. 2024Are LLMs good pragmatic speakers?cited by 7
  35. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learningcited by 6
  36. 2021A pragmatic account of the weak evidence effectcited by 2
  37. 2025Wild posteriors in the wildcited by 2
  38. 2025Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026When Do LLM Preferences Predict Downstream Behavior?cited by 7
  2. 2025On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Conceptscited by 5
  3. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Models Inherit Value Biases from Pretraining
  2. 2026Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models
  3. 2026Alignment Makes Language Models Normative, Not Descriptive
  4. 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
  5. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  6. 2025Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfare
  7. 2026Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations
  8. 2025Human Preferences for Constructive Interactions in Language Model Alignment