# Bob and Alice Go to a Bar: Reasoning About Future With Probabilistic Programs

arXiv: https://arxiv.org/abs/2108.03834 · HTML: https://mlsanity.com/paper/2108.03834/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 7 of 7)

- [An Introduction to Probabilistic Programming](https://mlsanity.com/paper/1809.10756/) (2018)
- [A General Multi-agent Epistemic Planner Based on Higher-order Belief Change](https://mlsanity.com/paper/1806.11298/) (2018)
- [Probabilistic Programs with Stochastic Conditioning](https://mlsanity.com/paper/2010.00282/) (2020)
- [Nested Reasoning About Autonomous Agents Using Probabilistic Programs](https://mlsanity.com/paper/1812.01569/) (2018)
- [Black-Box Policy Search with Probabilistic Programs](https://mlsanity.com/paper/1507.04635/) (2015)
- [Deployable probabilistic programming](https://mlsanity.com/paper/1906.11199/) (2019)
- [Nesting Probabilistic Programs](https://mlsanity.com/paper/1803.06328/) (2018)

## Abstract (arXiv)

It is well known that reinforcement learning can be cast as inference in an appropriate probabilistic model. However, this commonly involves introducing a distribution over agent trajectories with probabilities proportional to exponentiated rewards. In this work, we formulate reinforcement learning as Bayesian inference without resorting to rewards, and show that rewards are derived from agent's preferences, rather than the other way around. We argue that agent preferences should be specified stochastically rather than deterministically. Reinforcement learning via inference with stochastic preferences naturally describes agent behaviors, does not require introducing rewards and exponential weighing of trajectories, and allows to reason about agents using the solid foundation of Bayesian statistics. Stochastic conditioning, a probabilistic programming paradigm for conditioning models on distributions rather than values, is the formalism behind agents with probabilistic preferences. We demonstrate realization of our approach on case studies using both a two-agent coordinate game and a single agent acting in a noisy environment, showing that despite superficial differences, both cases can be modeled and reasoned about based on the same principles.