# Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets

arXiv: https://arxiv.org/abs/2301.08688 · HTML: https://mlsanity.com/paper/2301.08688/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Universal features of price formation in financial markets: perspectives from Deep Learning](https://mlsanity.com/paper/1803.06917/) (2018)
- [Double Deep Q-Learning for Optimal Execution](https://mlsanity.com/paper/1812.06600/) (2018)
- [ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation and Application to Financial Markets](https://mlsanity.com/paper/2110.14771/) (2021)
- [ABIDES: Towards High-Fidelity Market Simulation for AI Research](https://mlsanity.com/paper/1904.12066/) (2019)
- [Towards Realistic Market Simulations: a Generative Adversarial Networks Approach](https://mlsanity.com/paper/2110.13287/) (2021)
- [Learning to simulate realistic limit order book markets from data as a World Agent](https://mlsanity.com/paper/2210.09897/) (2022)
- [Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization](https://mlsanity.com/paper/1901.08740/) (2019)
- [Deep Reinforcement Learning for Active High Frequency Trading](https://mlsanity.com/paper/2101.07107/) (2021)
- [Model-based Reinforcement Learning for Predictions and Control for Limit Order Books](https://mlsanity.com/paper/1910.03743/) (2019)
- [Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units](https://mlsanity.com/paper/2105.10430/) (2021)
- [Deep Learning for Market by Order Data](https://mlsanity.com/paper/2102.08811/) (2021)

## Built on by (most cited first, 3 of 3)

- [Deep Limit Order Book Forecasting](https://mlsanity.com/paper/2403.09267/) (2024)
- [Optimal Execution with Reinforcement Learning](https://mlsanity.com/paper/2411.06389/) (2024)
- [FARE: Deep Reinforcement Learning For Fair Exposure Constrained Uncertainty Aware Financial Content Personalization](https://mlsanity.com/paper/2609.31890/) (2026)

## Abstract (arXiv)

We employ deep reinforcement learning (RL) to train an agent to successfully translate a high-frequency trading signal into a trading strategy that places individual limit orders. Based on the ABIDES limit order book simulator, we build a reinforcement learning OpenAI gym environment and utilise it to simulate a realistic trading environment for NASDAQ equities based on historic order book messages. To train a trading agent that learns to maximise its trading return in this environment, we use Deep Duelling Double Q-learning with the APEX (asynchronous prioritised experience replay) architecture. The agent observes the current limit order book state, its recent history, and a short-term directional forecast. To investigate the performance of RL for adaptive trading independently from a concrete forecasting algorithm, we study the performance of our approach utilising synthetic alpha signals obtained by perturbing forward-looking returns with varying levels of noise. Here, we find that the RL agent learns an effective trading strategy for inventory management and order placing that outperforms a heuristic benchmark trading strategy having access to the same signal.