# MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization

arXiv: https://arxiv.org/abs/2210.01774 · HTML: https://mlsanity.com/paper/2210.01774/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction](https://mlsanity.com/paper/1704.02971/) (2017)
- [A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem](https://mlsanity.com/paper/1706.10059/) (2017)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis](https://mlsanity.com/paper/1806.08946/) (2018)
- [Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction](https://mlsanity.com/paper/1712.02136/) (2017)
- [Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States](https://mlsanity.com/paper/2002.05780/) (2020)
- [AlphaStock: A Buying-Winners-and-Selling-Losers Investment Strategy using Interpretable Deep Reinforcement Attention Networks](https://mlsanity.com/paper/1908.02646/) (2019)
- [Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning](https://mlsanity.com/paper/1805.11240/) (2018)
- [Guided Meta-Policy Search](https://mlsanity.com/paper/1904.00956/) (2019)
- [Reinforcement Learning for Quantitative Trading](https://mlsanity.com/paper/2109.13851/) (2021)
- [Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution](https://mlsanity.com/paper/2012.12620/) (2020)
- [MAPS: Multi-agent Reinforcement Learning-based Portfolio Management System](https://mlsanity.com/paper/2007.05402/) (2020)
- [MSPM: A Modularized and Scalable Multi-Agent Reinforcement Learning-based System for Financial Portfolio Management](https://mlsanity.com/paper/2102.03502/) (2021)
- [Trade When Opportunity Comes: Price Movement Forecasting via Locality-Aware Attention and Iterative Refinement Labeling](https://mlsanity.com/paper/2107.11972/) (2021)

## Built on by (most cited first, 10 of 10)

- [FinCast: A Foundation Model for Financial Time-Series Forecasting](https://mlsanity.com/paper/2508.19609/) (2025)
- [From Deep Learning to LLMs: A survey of AI in Quantitative Investment](https://mlsanity.com/paper/2503.21422/) (2025)
- [MASS: Muli-agent simulation scaling for portfolio construction](https://mlsanity.com/paper/2505.10278/) (2025)
- [MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading](https://mlsanity.com/paper/2406.14537/) (2024)
- [IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making](https://mlsanity.com/paper/2308.08918/) (2023)
- [Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimization](https://mlsanity.com/paper/2403.07916/) (2024)
- [Enhancing Few-Shot Stock Trend Prediction with Large Language Models](https://mlsanity.com/paper/2407.09003/) (2024)
- [DeepClair: Utilizing Market Forecasts for Effective Portfolio Selection](https://mlsanity.com/paper/2407.13427/) (2024)
- [QuantBench: Benchmarking AI Methods for Quantitative Investment](https://mlsanity.com/paper/2504.18600/) (2025)
- [An Imitative Reinforcement Learning Framework for Pursuit-Lock-Launch Missions](https://mlsanity.com/paper/2406.11562/) (2024)

## Abstract (arXiv)

Portfolio management is a fundamental problem in finance. It involves periodic reallocations of assets to maximize the expected returns within an appropriate level of risk exposure. Deep reinforcement learning (RL) has been considered a promising approach to solving this problem owing to its strong capability in sequential decision making. However, due to the non-stationary nature of financial markets, applying RL techniques to portfolio optimization remains a challenging problem. Extracting trading knowledge from various expert strategies could be helpful for agents to accommodate the changing markets. In this paper, we propose MetaTrader, a novel two-stage RL-based approach for portfolio management, which learns to integrate diverse trading policies to adapt to various market conditions. In the first stage, MetaTrader incorporates an imitation learning objective into the reinforcement learning framework. Through imitating different expert demonstrations, MetaTrader acquires a set of trading policies with great diversity. In the second stage, MetaTrader learns a meta-policy to recognize the market conditions and decide on the most proper learned policy to follow. We evaluate the proposed approach on three real-world index datasets and compare it to state-of-the-art baselines. The empirical results demonstrate that MetaTrader significantly outperforms those baselines in balancing profits and risks. Furthermore, thorough ablation studies validate the effectiveness of the components in the proposed approach.