# FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

arXiv: https://arxiv.org/abs/2111.09395 · HTML: https://mlsanity.com/paper/2111.09395/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 53 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem](https://mlsanity.com/paper/1706.10059/) (2017)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Deep Hedging](https://mlsanity.com/paper/1802.03042/) (2018)
- [Deep Reinforcement Learning for Trading](https://mlsanity.com/paper/1911.10107/) (2019)
- [Practical Deep Reinforcement Learning Approach for Stock Trading](https://mlsanity.com/paper/1811.07522/) (2018)
- [Experience Replay Optimization](https://mlsanity.com/paper/1906.08387/) (2019)
- [Get Real: Realism Metrics for Robust Limit Order Book Market Simulations](https://mlsanity.com/paper/1912.04941/) (2019)
- [Reinforcement Learning for Market Making in a Multi-agent Dealer Market](https://mlsanity.com/paper/1911.05892/) (2019)
- [Cryptocurrency Portfolio Management with Deep Reinforcement Learning](https://mlsanity.com/paper/1612.01277/) (2016)
- [Financial Trading as a Game: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1807.02787/) (2018)
- [Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis](https://mlsanity.com/paper/1906.11046/) (2019)
- [TensorLayer: A Versatile Library for Efficient Deep Learning Development](https://mlsanity.com/paper/1707.08551/) (2017)
- [Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertainty](https://mlsanity.com/paper/2006.12686/) (2020)
- [Sentiment and Knowledge Based Algorithmic Trading with Deep Reinforcement Learning](https://mlsanity.com/paper/2001.09403/) (2020)

## Built on by (most cited first, 30 of 58)

- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023)
- [FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making](https://mlsanity.com/paper/2407.06567/) (2024)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [FinGPT: Democratizing Internet-scale Data for Financial Large Language Models](https://mlsanity.com/paper/2307.10485/) (2023)
- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay](https://mlsanity.com/paper/2502.16789/) (2025)
- [FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting](https://mlsanity.com/paper/2502.18834/) (2025)
- [FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading](https://mlsanity.com/paper/2502.11433/) (2025)
- [Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?](https://mlsanity.com/paper/2505.07078/) (2025)
- [Efficient Integration of Multi-Order Dynamics and Internal Dynamics in Stock Movement Prediction](https://mlsanity.com/paper/2211.07400/) (2022)
- [Dynamic Datasets and Market Environments for Financial Reinforcement Learning](https://mlsanity.com/paper/2304.13174/) (2023)
- [Cognitive Alpha Mining via LLM-Driven Code-Based Evolution](https://mlsanity.com/paper/2511.18850/) (2025)
- [CFGPT: Chinese Financial Assistant with Large Language Model](https://mlsanity.com/paper/2309.10654/) (2023)
- [Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective](https://mlsanity.com/paper/2305.18464/) (2023)
- [FinRL-Meta: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance](https://mlsanity.com/paper/2112.06753/) (2021)
- [Optimizing Trading Strategies in Quantitative Markets using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.11959/) (2023)
- [Can we hop in general? A discussion of benchmark selection and design using the Hopper environment](https://mlsanity.com/paper/2410.08870/) (2024)
- [In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates](https://mlsanity.com/paper/2410.15612/) (2024)
- [Adaptive Liquidity Provision in Uniswap V3 with Deep Reinforcement Learning](https://mlsanity.com/paper/2309.10129/) (2023)
- [Combining Transformer based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization](https://mlsanity.com/paper/2402.16609/) (2024)
- [MILLION: A General Multi-Objective Framework with Controllable Risk for Portfolio Management](https://mlsanity.com/paper/2412.03038/) (2024)
- [Deep Reinforcement Learning for Cryptocurrency Trading: Practical Approach to Address Backtest Overfitting](https://mlsanity.com/paper/2209.05559/) (2022)
- [XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library](https://mlsanity.com/paper/2312.16248/) (2023)
- [Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Information](https://mlsanity.com/paper/2401.04729/) (2024)
- [Are Large Language Models Good In-context Learners for Financial Sentiment Analysis?](https://mlsanity.com/paper/2503.04873/) (2025)
- [Explainable Post hoc Portfolio Management Financial Policy of a Deep Reinforcement Learning agent](https://mlsanity.com/paper/2407.14486/) (2024)
- [Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning](https://mlsanity.com/paper/2507.00257/) (2025)
- [Language Model Guided Reinforcement Learning in Quantitative Trading](https://mlsanity.com/paper/2508.02366/) (2025)
- [Deep Reinforcement Learning for ESG financial portfolio management](https://mlsanity.com/paper/2307.09631/) (2023)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely textitto decide where to trade, at what price and textitwhat quantity, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework textitFinRL as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, textitfull-stack framework, customization, reproducibility and textithands-on tutoring. Embodied as a three-layer architecture with modular structures, FinRL implements fine-tuned state-of-the-art DRL algorithms and common reward functions, while alleviating the debugging workloads. Thus, we help users pipeline the strategy design at a high turnover rate. At multiple levels of time granularity, FinRL simulates various markets as training environments using historical data and live trading APIs. Being highly extensible, FinRL reserves a set of user-import interfaces and incorporates trading constraints such as market friction, market liquidity and investor's risk-aversion. Moreover, serving as practitioners' stepping stones, typical trading tasks are provided as step-by-step tutorials, e.g., stock trading, portfolio allocation, cryptocurrency trading, etc.