mlsanityData to 7 Oct 2026

Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization

arXiv 2505.12759 ↗First posted 19 May 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data. However, traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset. These offline policies are less generalizable as they fail to account for the non-stationary nature of the market. Our approach, MetaTrader, frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions. First, MetaTrader employs a bilevel learning framework that explicitly trains the RL agent to improve both in-domain profits on the original dataset and out-of-domain performance across diverse transformations of the raw financial data. Second, our approach incorporates a new temporal difference (TD) method that approximates worst-case TD estimates from a batch of transformed TD targets, addressing the value overestimation issue that is particularly challenging in scenarios with limited offline data. Our empirical results on two public stock datasets show that MetaTrader outperforms existing methods, including both RL-based approaches and traditional stock prediction models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

32 of 46 references.

References · 32

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  8. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  9. 2018How to train your MAMLcited by 306
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  11. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
20 more
  1. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  2. 2019Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learningcited by 121
  3. 2018Temporal Relational Ranking for Stock Predictioncited by 100
  4. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  5. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  6. 2019Meta reinforcement learning as task inferencecited by 75
  7. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 57
  8. 2020Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction Statescited by 55
  9. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  10. 2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
  11. 2023MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learningcited by 35
  12. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Modelcited by 31
  13. 2021Deep Reinforcement Learning for Active High Frequency Tradingcited by 21
  14. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 17
  15. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  16. 2023Train Hard, Fight Easy: Robust Meta Reinforcement Learningcited by 11
  17. 2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptationcited by 10
  18. 2021Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performancecited by 7
  19. 2021Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learningcited by 5
  20. 2021Attentional Meta-learners for Few-shot Polythetic Classificationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization
  2. 2023Towards Generalizable Reinforcement Learning for Trade Execution
  3. 2025Diverse Transformer Decoding for Offline Reinforcement Learning Using Financial Algorithmic Approaches
  4. 2022Wall Street Tree Search: Risk-Aware Planning for Offline Reinforcement Learning
  5. 2024Robot See, Robot Do: Imitation Reward for Noisy Financial Environments
  6. 2023An Offline Learning Approach to Propagator Models
  7. 2023PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets
  8. 2022Combining Reinforcement Learning and Inverse Reinforcement Learning for Asset Allocation Recommendations