# Model-based Reinforcement Learning for Predictions and Control for Limit Order Books

arXiv: https://arxiv.org/abs/1910.03743 · HTML: https://mlsanity.com/paper/1910.03743/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 14 of 14)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Towards Realistic Market Simulations: a Generative Adversarial Networks Approach](https://mlsanity.com/paper/2110.13287/) (2021)
- [A Review of Reinforcement Learning in Financial Applications](https://mlsanity.com/paper/2411.12746/) (2024)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units](https://mlsanity.com/paper/2105.10430/) (2021)
- [Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets](https://mlsanity.com/paper/2301.08688/) (2023)
- [Microstructure-Empowered Stock Factor Extraction and Utilization](https://mlsanity.com/paper/2308.08135/) (2023)
- [Market Making with Deep Reinforcement Learning from Limit Order Books](https://mlsanity.com/paper/2305.15821/) (2023)
- [Optimistic Regret Bounds for Online Learning in Adversarial Markov Decision Processes](https://mlsanity.com/paper/2405.02188/) (2024)
- [Pretraining with random noise for uncertainty calibration](https://mlsanity.com/paper/2412.17411/) (2024)
- [Deep Reinforcement Learning for Optimal Asset Allocation Using DDPG with TiDE](https://mlsanity.com/paper/2508.20103/) (2025)
- [Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning](https://mlsanity.com/paper/2412.17908/) (2024)
- [The Invisible Handshake: Persistent Overpricing by Adaptive Market Agents](https://mlsanity.com/paper/2510.15995/) (2025)
- [DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading](https://mlsanity.com/paper/2605.25527/) (2026)

## Abstract (arXiv)

We build a profitable electronic trading agent with Reinforcement Learning that places buy and sell orders in the stock market. An environment model is built only with historical observational data, and the RL agent learns the trading policy by interacting with the environment model instead of with the real-market to minimize the risk and potential monetary loss. Trained in unsupervised and self-supervised fashion, our environment model learned a temporal and causal representation of the market in latent space through deep neural networks. We demonstrate that the trading policy trained entirely within the environment model can be transferred back into the real market and maintain its profitability. We believe that this environment model can serve as a robust simulator that predicts market movement as well as trade impact for further studies.