mlsanityData to 7 Oct 2026

Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets

arXiv 2510.19950 ↗First posted 22 Oct 2025cs.LG cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact. This mismatch between training and deployment environments can significantly degrade performance. Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties, but typically rely on symmetric structures that fail to capture the directional nature of market impact. To address this issue, we develop a novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 91 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

43 of 91 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Convex optimizationcited by 2,479
  3. 2017Robust Adversarial Reinforcement Learningcited by 439
  4. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  5. 2024Reinforcement learningcited by 236
  6. 2018Deep Hedgingcited by 106
  7. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  8. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  9. 2022Robust Reinforcement Learning using Offline Datacited by 95
  10. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  11. 2019Deep Reinforcement Learning for Tradingcited by 73
  12. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
31 more
  1. 2019Wasserstein Robust Reinforcement Learningcited by 50
  2. 2019Distributionally Robust Reinforcement Learningcited by 40
  3. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  4. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  5. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  6. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  7. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 28
  8. 2024MarS: a Financial Market Simulation Engine Powered by Generative Foundation Modelcited by 27
  9. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  10. 2024Simulating Financial Market via Large Language Model based Agentscited by 25
  11. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  12. 2019Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimizationcited by 22
  13. 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
  14. 2021Robust Risk-Aware Reinforcement Learningcited by 17
  15. 2023Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learningcited by 16
  16. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  17. 2019Reinforcement Learning for Portfolio Managementcited by 12
  18. 2020Variance-Reduced Off-Policy TDC Learning: Non-Asymptotic Convergence Analysiscited by 12
  19. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
  20. 2024The Evolution of Reinforcement Learning in Quantitative Finance: A Surveycited by 11
  21. 2023Robust Risk-Aware Option Hedgingcited by 9
  22. 2025Provably Robust Federated Reinforcement Learningcited by 8
  23. 2021Greedy-GQ with Variance Reduction: Finite-time Analysis and Improved Complexitycited by 7
  24. 2023Rectangularity and duality of distributionally robust Markov Decision Processescited by 6
  25. 2023An Efficient Solution to s-Rectangular Robust Markov Decision Processescited by 5
  26. 2024Developing A Multi-Agent and Self-Adaptive Framework with Deep Reinforcement Learning for Dynamic Portfolio Risk Managementcited by 5
  27. 2025TRADES: Generating Realistic Market Simulations with Diffusion Modelscited by 5
  28. 2024Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimizationcited by 4
  29. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  30. 2023Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertaintycited by 2
  31. 2020Empirical Study of Market Impact Conditional on Order-Flow Imbalancecited by 1

Built on by

Not cited yet.