# Imitation Learning of Factored Multi-agent Reactive Models

arXiv: https://arxiv.org/abs/1903.04714 · HTML: https://mlsanity.com/paper/1903.04714/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 12 Mar 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 21 of 21)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Variational inference for Monte Carlo objectives](https://mlsanity.com/paper/1602.06725/) (2016)
- [REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models](https://mlsanity.com/paper/1703.07370/) (2017)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [Reweighted Wake-Sleep](https://mlsanity.com/paper/1406.2751/) (2014)
- [Unsupervised Perceptual Rewards for Imitation Learning](https://mlsanity.com/paper/1612.06699/) (2016)
- [Emergence of grid-like representations by training recurrent neural networks to perform spatial localization](https://mlsanity.com/paper/1803.07770/) (2018)
- [Generating Multi-Agent Trajectories using Programmatic Weak Supervision](https://mlsanity.com/paper/1803.07612/) (2018)
- [Generative Models of Visually Grounded Imagination](https://mlsanity.com/paper/1705.10762/) (2017)
- [Learning recurrent representations for hierarchical behavior modeling](https://mlsanity.com/paper/1611.00094/) (2016)
- [Generative Temporal Models with Memory](https://mlsanity.com/paper/1702.04649/) (2017)
- [Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow](https://mlsanity.com/paper/1805.10469/) (2018)

## Abstract (arXiv)

We apply recent advances in deep generative modeling to the task of imitation learning from biological agents. Specifically, we apply variations of the variational recurrent neural network model to a multi-agent setting where we learn policies of individual uncoordinated agents acting based on their perceptual inputs and their hidden belief state. We learn stochastic policies for these agents directly from observational data, without constructing a reward function. An inference network learned jointly with the policy allows for efficient inference over the agent's belief state given a sequence of its current perceptual inputs and the prior actions it performed, which lets us extrapolate observed sequences of behavior into the future while maintaining uncertainty estimates over future trajectories. We test our approach on a dataset of flies interacting in a 2D environment, where we demonstrate better predictive performance than existing approaches which learn deterministic policies with recurrent neural networks. We further show that the uncertainty estimates over future trajectories we obtain are well calibrated, which makes them useful for a variety of downstream processing tasks.