mlsanityData to 14 Oct 2026

Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions

arXiv 2505.11614 ↗First posted 16 May 2025cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

A central goal of cognitive modeling is to develop models that not only predict human behavior but also provide insight into the underlying cognitive mechanisms. While neural network models trained on large-scale behavioral data often achieve strong predictive performance, they typically fall short in offering interpretable explanations of the cognitive processes they capture. In this work, we explore the potential of pretrained large language models (LLMs) to serve as dual-purpose cognitive models--capable of both accurate prediction and interpretable explanation in natural language. Specifically, we employ reinforcement learning with outcome-based rewards to guide LLMs toward generating explicit reasoning traces for explaining human risky choices. Our findings demonstrate that this approach produces high-quality explanations alongside strong quantitative predictions of human decisions.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 24 of its 56 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

24 of 56 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  11. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  12. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
12 more
  1. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  2. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  3. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  4. 2022Impossibility Theorems for Feature Attributioncited by 78
  5. 2023Turning large language models into cognitive modelscited by 67
  6. 2025Reinforcement Learning from Human Feedbackcited by 58
  7. 2024Centaur: a foundation model of human cognitioncited by 36
  8. 2023Bayes in the age of intelligent machinescited by 22
  9. 2024Capturing the Complexity of Human Strategic Decision-Making with Machine Learningcited by 14
  10. 2024Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Gamescited by 12
  11. 2024Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choicecited by 12
  12. 2023Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Finetuning LLMs for Human Behavior Prediction in Social Science Experimentscited by 39
  2. 2026HumanLM: Simulating Users with State Alignment Beats Response Imitationcited by 37
  3. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  4. 2026Learning Next Action Predictors from Human-Computer Interactioncited by 7
  5. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 7
  6. 2025Addressing Longstanding Challenges in Cognitive Science with Language Modelscited by 3
  7. 2026Small Foundation Models of Human Cognition and Behaviourcited by 1
  8. 2026LLMs Should Not Yet Be Credited with Decision Explanationcited by 0
  9. 2026Position: We Need Practical AI Alignment Methods to Mirror Human Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games
  2. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  3. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions
  4. 2026Calibrating LLM Explanations to Human Decisions with Reinforcement Learning
  5. 2026Understanding Large Language Models
  6. 2024Dual Traits in Probabilistic Reasoning of Large Language Models
  7. 2024Mental Modeling of Reinforcement Learning Agents by Language Models
  8. 2026Reasoning aligns language models to human cognition