mlsanityData to 14 Oct 2026

Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective

arXiv 2305.17760 ↗First posted 28 May 2023cs.CL cs.LGcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

How do language models "think"? This paper formulates a probabilistic cognitive model called the bounded pragmatic speaker, which can characterize the operation of different variations of language models. Specifically, we demonstrate that large language models fine-tuned with reinforcement learning from human feedback (Ouyang et al., 2022) embody a model of thought that conceptually resembles a fast-and-slow model (Kahneman, 2011), which psychologists have attributed to humans. We discuss the limitations of reinforcement learning from human feedback as a fast-and-slow model of thought and propose avenues for expanding this framework. In essence, our research highlights the value of adopting a cognitive probabilistic modeling approach to gain insights into the comprehension, evaluation, and advancement of language models.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 61 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

39 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
26 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  9. 2021Causal Abstractions of Neural Networkscited by 422
  10. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  11. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  12. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  13. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 185
  14. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
  15. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  16. 2022Language Models as Agent Modelscited by 132
  17. 2022Coder Reviewer Reranking for Code Generationcited by 84
  18. 2016Reasoning About Pragmatics with Neural Listeners and Speakerscited by 68
  19. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  20. 2021Interactive Learning from Activity Descriptioncited by 26
  21. 2021Innovative Bert-based Reranking Language Models for Speech Recognitioncited by 23
  22. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  23. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  24. 2020Learning to refer informatively by amortizing pragmatic reasoningcited by 17
  25. 2021Calibrate your listeners! Robust communication-based training for pragmatic speakerscited by 9
  26. 2022Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 2.

10 papers.

  1. 2023Cognitive Architectures for Language Agentscited by 429
  2. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 49
  3. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
  4. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  5. 2024Are LLMs good pragmatic speakers?cited by 7
  6. 2025Distinguishing AI-Generated and Human-Written Text Through Psycholinguistic Analysiscited by 3
  7. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  8. 2026CRAFT: Grounded Multi-Agent Coordination Under Partial Informationcited by 3
  9. 2024Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions?cited by 0
  10. 2026Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual Traits in Probabilistic Reasoning of Large Language Models
  2. 2026Reasoning aligns language models to human cognition
  3. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  4. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates
  5. 2025The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models
  6. 2026Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity
  7. 2025On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
  8. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions