# Double Deep Q-Learning for Optimal Execution

arXiv: https://arxiv.org/abs/1812.06600 · HTML: https://mlsanity.com/paper/1812.06600/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2018.

## Built on by (most cited first, 30 of 43)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Universal Trading for Order Execution with Oracle Policy Distillation](https://mlsanity.com/paper/2103.10860/) (2021)
- [Reinforcement Learning for Quantitative Trading](https://mlsanity.com/paper/2109.13851/) (2021)
- [Conditional Generators for Limit Order Book Environments: Explainability, Challenges, and Robustness](https://mlsanity.com/paper/2306.12806/) (2023)
- [JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading](https://mlsanity.com/paper/2308.13289/) (2023)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [From Deep Learning to LLMs: A survey of AI in Quantitative Investment](https://mlsanity.com/paper/2503.21422/) (2025)
- [Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance](https://mlsanity.com/paper/2307.03119/) (2023)
- [Towards Generalizable Reinforcement Learning for Trade Execution](https://mlsanity.com/paper/2307.11685/) (2023)
- [Optimal Execution with Reinforcement Learning](https://mlsanity.com/paper/2411.06389/) (2024)
- [Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution](https://mlsanity.com/paper/2207.11152/) (2022)
- [Deep Hedging with Market Impact](https://mlsanity.com/paper/2402.13326/) (2024)
- [Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets](https://mlsanity.com/paper/2107.08083/) (2021)
- [Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets](https://mlsanity.com/paper/2301.08688/) (2023)
- [Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network](https://mlsanity.com/paper/2206.14267/) (2022)
- [Q-Learning under Finite Model Uncertainty](https://mlsanity.com/paper/2407.04259/) (2024)
- [Deep Reinforcement Learning for Online Optimal Execution Strategies](https://mlsanity.com/paper/2410.13493/) (2024)
- [Safe and Compliant Cross-Market Trade Execution via Constrained RL and Zero-Knowledge Audits](https://mlsanity.com/paper/2510.04952/) (2025)
- [Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution](https://mlsanity.com/paper/2511.15262/) (2025)
- [Learning in Markov Decision Processes with Exogenous Dynamics](https://mlsanity.com/paper/2603.02862/) (2026)
- [Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Execution](https://mlsanity.com/paper/2605.20348/) (2026)
- [Imitate then Transcend: Multi-Agent Optimal Execution with Dual-Window Denoise PPO](https://mlsanity.com/paper/2206.10736/) (2022)
- [A Modular Framework for Reinforcement Learning Optimal Execution](https://mlsanity.com/paper/2208.06244/) (2022)
- [Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization](https://mlsanity.com/paper/2212.14670/) (2022)
- [Optimal Execution Using Reinforcement Learning](https://mlsanity.com/paper/2306.17178/) (2023)
- [Deviations from the Nash equilibrium in a two-player optimal execution game with reinforcement learning](https://mlsanity.com/paper/2408.11773/) (2024)
- [Trading through Earnings Seasons using Self-Supervised Contrastive Representation Learning](https://mlsanity.com/paper/2409.17392/) (2024)
- [Deep reinforcement learning for optimal trading with partial information](https://mlsanity.com/paper/2511.00190/) (2025)
- [Learning Market Making with Closing Auctions](https://mlsanity.com/paper/2601.17247/) (2026)
- [Reinforcement Learning for Financial Index Tracking](https://mlsanity.com/paper/2308.02820/) (2023)

## Abstract (arXiv)

Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios.