# Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network

arXiv: https://arxiv.org/abs/2206.14267 · HTML: https://mlsanity.com/paper/2206.14267/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 28 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance](https://mlsanity.com/paper/2011.09607/) (2020)
- [Deep Reinforcement Learning for Trading](https://mlsanity.com/paper/1911.10107/) (2019)
- [Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States](https://mlsanity.com/paper/2002.05780/) (2020)
- [Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon](https://mlsanity.com/paper/2011.10300/) (2020)
- [Discount Factor as a Regularizer in Reinforcement Learning](https://mlsanity.com/paper/2007.02040/) (2020)
- [Double Deep Q-Learning for Optimal Execution](https://mlsanity.com/paper/1812.06600/) (2018)
- [Financial Trading as a Game: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1807.02787/) (2018)
- [Data Normalization for Bilinear Structures in High-Frequency Financial Time-series](https://mlsanity.com/paper/2003.00598/) (2020)
- [On the Convergence of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning](https://mlsanity.com/paper/2002.03585/) (2020)
- [Low-rank State-action Value-function Approximation](https://mlsanity.com/paper/2104.08805/) (2021)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Beyond Exponentially Discounted Sum: Automatic Learning of Return Function](https://mlsanity.com/paper/1905.11591/) (2019)
- [Designing Rewards for Fast Learning](https://mlsanity.com/paper/2205.15400/) (2022)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [A Dynamic Programming Algorithm for Finding an Optimal Sequence of Informative Measurements](https://mlsanity.com/paper/2109.11808/) (2021)
- [High-Dimensional Stock Portfolio Trading with Deep Reinforcement Learning](https://mlsanity.com/paper/2112.04755/) (2021)
- [Extracting Common Time Trends from Concurrent Time Series: Maximum Autocorrelation Factors with Application to Tree Ring Time Series Data](https://mlsanity.com/paper/1502.01073/) (2015)
- [Deep Reinforcement Learning for Portfolio Optimization using Latent Feature State Space (LFSS) Module](https://mlsanity.com/paper/2102.06233/) (2021)
- [Deep Reinforcement Learning, a textbook](https://mlsanity.com/paper/2201.02135/) (2022)
- [Financial Vision Based Reinforcement Learning Trading Strategy](https://mlsanity.com/paper/2202.04115/) (2022)
- [A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning](https://mlsanity.com/paper/2206.02034/) (2022)

## Built on by (most cited first, 2 of 2)

- [Dueling Deep Reinforcement Learning for Financial Time Series](https://mlsanity.com/paper/2504.11601/) (2025)
- [The Enhanced Physics-Informed Kolmogorov-Arnold Networks: Applications of Newton's Laws in Financial Deep Reinforcement Learning (RL) Algorithms](https://mlsanity.com/paper/2602.01388/) (2026)

## Abstract (arXiv)

This paper presents a Double Deep Q-Network algorithm for trading single assets, namely the E-mini S&P 500 continuous futures contract. We use a proven setup as the foundation for our environment with multiple extensions. The features of our trading agent are constantly being expanded to include additional assets such as commodities, resulting in four models. We also respond to environmental conditions, including costs and crises. Our trading agent is first trained for a specific time period and tested on new data and compared with the long-and-hold strategy as a benchmark (market). We analyze the differences between the various models and the in-sample/out-of-sample performance with respect to the environment. The experimental results show that the trading agent follows an appropriate behavior. It can adjust its policy to different circumstances, such as more extensive use of the neutral position when trading costs are present. Furthermore, the net asset value exceeded that of the benchmark, and the agent outperformed the market in the test set. We provide initial insights into the behavior of an agent in a financial domain using a DDQN algorithm. The results of this study can be used for further development.