# Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution

arXiv: https://arxiv.org/abs/2012.12620 · HTML: https://mlsanity.com/paper/2012.12620/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2020.

## Built on by (most cited first, 20 of 20)

- [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024)
- [FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making](https://mlsanity.com/paper/2407.06567/) (2024)
- [Reinforcement Learning for Quantitative Trading](https://mlsanity.com/paper/2109.13851/) (2021)
- [DeepScalper: A Risk-Aware Reinforcement Learning Framework to Capture Fleeting Intraday Trading Opportunities](https://mlsanity.com/paper/2201.09058/) (2021)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization](https://mlsanity.com/paper/2210.01774/) (2022)
- [Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance](https://mlsanity.com/paper/2307.03119/) (2023)
- [Reinforcement Learning with Maskable Stock Representation for Portfolio Management in Customizable Stock Pools](https://mlsanity.com/paper/2311.10801/) (2023)
- [MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading](https://mlsanity.com/paper/2406.14537/) (2024)
- [MILLION: A General Multi-Objective Framework with Controllable Risk for Portfolio Management](https://mlsanity.com/paper/2412.03038/) (2024)
- [PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets](https://mlsanity.com/paper/2302.00586/) (2023)
- [A novel multi-agent dynamic portfolio optimization learning system based on hierarchical deep reinforcement learning](https://mlsanity.com/paper/2501.06832/) (2025)
- [Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization](https://mlsanity.com/paper/2212.14670/) (2022)
- [Market Making with Deep Reinforcement Learning from Limit Order Books](https://mlsanity.com/paper/2305.15821/) (2023)
- [MARS: A Meta-Adaptive Reinforcement Learning Framework for Risk-Aware Multi-Agent Portfolio Management](https://mlsanity.com/paper/2508.01173/) (2025)
- [Cryptocurrency Portfolio Optimization by Neural Networks](https://mlsanity.com/paper/2310.01148/) (2023)
- [Dynamic Portfolio Optimization via Augmented DDPG with Quantum Price Levels-Based Trading Strategy](https://mlsanity.com/paper/2501.08528/) (2025)
- [Bayesian Robust Financial Trading with Adversarial Synthetic Market Data](https://mlsanity.com/paper/2601.17008/) (2026)
- [Regime-Adaptive Continual Learning for Portfolio Management](https://mlsanity.com/paper/2606.00143/) (2026)
- [Can Large Language Models Execute Parent Orders?](https://mlsanity.com/paper/2607.28410/) (2026)

## Abstract (arXiv)

Portfolio management via reinforcement learning is at the forefront of fintech research, which explores how to optimally reallocate a fund into different financial assets over the long term by trial-and-error. Existing methods are impractical since they usually assume each reallocation can be finished immediately and thus ignoring the price slippage as part of the trading cost. To address these issues, we propose a hierarchical reinforced stock trading system for portfolio management (HRPM). Concretely, we decompose the trading process into a hierarchy of portfolio management over trade execution and train the corresponding policies. The high-level policy gives portfolio weights at a lower frequency to maximize the long term profit and invokes the low-level policy to sell or buy the corresponding shares within a short time window at a higher frequency to minimize the trading cost. We train two levels of policies via pre-training scheme and iterative training scheme for data efficiency. Extensive experimental results in the U.S. market and the China market demonstrate that HRPM achieves significant improvement against many state-of-the-art approaches.