# Towards Inverse Reinforcement Learning for Limit Order Book Dynamics

arXiv: https://arxiv.org/abs/1906.04813 · HTML: https://mlsanity.com/paper/1906.04813/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 11 Jun 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 8 of 8)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Learning Deep Mean Field Games for Modeling Large Population Behavior](https://mlsanity.com/paper/1711.03156/) (2017)
- [Inverse Reinforcement Learning via Deep Gaussian Process](https://mlsanity.com/paper/1512.08065/) (2015)
- [Inferring agent objectives at different scales of a complex adaptive system](https://mlsanity.com/paper/1712.01137/) (2017)

## Built on by (most cited first, 2 of 2)

- [A Review of Reinforcement Learning in Financial Applications](https://mlsanity.com/paper/2411.12746/) (2024)
- [Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2608.15770/) (2026)

## Abstract (arXiv)

Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquiring such reward functions through inference, allowing to generalize the resulting policy to states not observed in the past. This paper investigates whether IRL can infer such rewards from agents within real financial stochastic environments: limit order books (LOB). We introduce a simple one-level LOB, where the interactions of a number of stochastic agents and an expert trading agent are modelled as a Markov decision process. We consider two cases for the expert's reward: either a simple linear function of state features; or a complex, more realistic non-linear function. Given the expert agent's demonstrations, we attempt to discover their strategy by modelling their latent reward function using linear and Gaussian process (GP) regressors from previous literature, and our own approach through Bayesian neural networks (BNN). While the three methods can learn the linear case, only the GP-based and our proposed BNN methods are able to discover the non-linear reward case. Our BNN IRL algorithm outperforms the other two approaches as the number of samples increases. These results illustrate that complex behaviours, induced by non-linear reward functions amid agent-based stochastic scenarios, can be deduced through inference, encouraging the use of inverse reinforcement learning for opponent-modelling in multi-agent systems.