mlsanityData to 7 Oct 2026

The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models

arXiv 2204.10759 ↗First posted 22 Apr 2022cs.AI cs.LG cs.MAcited by 31

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Models of human behavior for prediction and collaboration tend to fall into two categories: ones that learn from large amounts of data via imitation learning, and ones that assume human behavior to be noisily-optimal for some reward function. The former are very useful, but only when it is possible to gather a lot of human data in the target environment and distribution. The advantage of the latter type, which includes Boltzmann rationality, is the ability to make accurate predictions in new environments without extensive data when humans are actually close to optimal. However, these models fail when humans exhibit systematic suboptimality, i.e. when their deviations from optimal behavior are not independent, but instead consistent over time. Our key insight is that systematic suboptimality can be modeled by predicting policies, which couple action choices over time, instead of trajectories. We introduce the Boltzmann policy distribution (BPD), which serves as a prior over human policies and adapts via Bayesian inference to capture systematic deviations by observing human actions during a single episode. The BPD is difficult to compute and represent because policies lie in a high-dimensional continuous space, but we leverage tools from generative and sequence models to enable efficient sampling and inference. We show that the BPD enables prediction of human behavior and human-AI collaboration equally as well as imitation learning-based human models while using far less data.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 49 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2016Variational Inference: A Review for Statisticianscited by 1,466
  11. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  12. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
15 more
  1. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  2. 2017One-Shot Imitation Learningcited by 232
  3. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  4. 2021Collaborating with Humans without Human Datacited by 139
  5. 2021Off-Belief Learningcited by 72
  6. 2017Multimodal Probabilistic Model-Based Planning for Human-Robot Interactioncited by 50
  7. 2015Shared Autonomy via Hindsight Optimizationcited by 46
  8. 2019Imitation Learning for Human Pose Predictioncited by 44
  9. 2017Variational Inference using Implicit Distributionscited by 39
  10. 2021A New Formalism, Method and Open Issues for Zero-Shot Coordinationcited by 31
  11. 2018Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learningcited by 28
  12. 2021Evaluating the Robustness of Collaborative Agentscited by 25
  13. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  14. 2017Optimal rates of entropy estimation over Lipschitz ballscited by 15
  15. 2021On complementing end-to-end human behavior predictors with planningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 6; 2025: 7; 2026: 6.

31 papers.

Sort:

By area:

  1. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  2. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  3. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  4. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  5. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  6. 2023Learning to Make Adherence-Aware Advicecited by 13
  7. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  8. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  9. 2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
  10. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 9
  11. 2023Efficient Human-AI Coordination via Preparatory Language-based Conventioncited by 7
  12. 2025MILE: Model-based Intervention Learningcited by 7
19 more
  1. 2023Concept Alignment as a Prerequisite for Value Alignmentcited by 5
  2. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  3. 2024Observation Interference in Partially Observable Assistance Gamescited by 4
  4. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decompositioncited by 3
  5. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learningcited by 3
  6. 2022Modeling Mobile Health Users as Reinforcement Learning Agentscited by 2
  7. 2023Learning to Influence Human Behavior with Offline Reinforcement Learningcited by 2
  8. 2023A behavioural transformer for effective collaboration between a robot and a non-stationary humancited by 2
  9. 2026Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamworkcited by 2
  10. 2022Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learningcited by 1
  11. 2023Quantifying Assistive Robustness Via the Natural-Adversarial Frontiercited by 1
  12. 2025Probabilistic Quantum SVM Training on Ising Machinecited by 1
  13. 2026Enhancing Goal Inference via Correction Timingcited by 1
  14. 2024Boltzmann State-Dependent Rationalitycited by 0
  15. 2025Supervised Reward Inferencecited by 0
  16. 2025Multi-Agent Inverse Q-Learning from Demonstrationscited by 0
  17. 2026On the Strengths and Weaknesses of Data for Open-set Embodied Assistancecited by 0
  18. 2026Learning the Preferences of a Learning Agentcited by 0
  19. 2026Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2023Learning Complementary Policies for Human-AI Teams
  3. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift
  4. 2026Mitigating Cognitive Bias in RLHF by Altering Rationality
  5. 2024How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation
  6. 2025Reinforcement Learning from Human Feedback
  7. 2022Misspecification in Inverse Reinforcement Learning
  8. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations