mlsanityData to 14 Oct 2026

Two Kinds of Learning Algorithms for Continuous-Time VWAP Targeting Execution

arXiv 2411.06645 ↗First posted 11 Nov 2024math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The optimal execution problem has always been a continuously focused research issue, and many reinforcement learning (RL) algorithms have been studied. In this article, we consider the execution problem of targeting the volume weighted average price (VWAP) and propose a relaxed stochastic optimization problem with an entropy regularizer to encourage more exploration. We derive the explicit formula of the optimal policy, which is Gaussian distributed, with its mean value being the solution to the original problem. Extending the framework of continuous RL to processes with jumps, we provide some theoretical proofs for RL algorithms. First, minimizing the martingale loss function leads to the optimal parameter estimates in the mean-square sense, and the second algorithm is to use the martingale orthogonality condition. In addition to the RL algorithm, we also propose another learning algorithm: adaptive dynamic programming (ADP) algorithm, and verify the performance of both in two different environments across different random seeds. Convergence of all algorithms has been verified in different environments, and shows a larger advantage in the environment with stronger price impact. ADP is a good choice when the agent fully understands the environment and can estimate the parameters well. On the other hand, RL algorithms do not require any model assumptions or parameter estimation, and are able to learn directly from interactions with the environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 75 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

18 of 75 references.

References · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Deep Hedgingcited by 108
  3. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  4. 2022q-Learning in Continuous Timecited by 91
  5. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  6. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  7. 2018Double Deep Q-Learning for Optimal Executioncited by 43
  8. 2019AlphaStock: A Buying-Winners-and-Selling-Losers Investment Strategy using Interpretable Deep Reinforcement Attention Networkscited by 40
  9. 2021Universal Trading for Order Execution with Oracle Policy Distillationcited by 30
  10. 2021Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Gamescited by 25
  11. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  12. 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
6 more
  1. 2018Exploration versus exploitation in reinforcement learning: a stochastic control approachcited by 18
  2. 2020Exploratory LQG Mean Field Games with Entropy Regularizationcited by 15
  3. 2020Stochastic Linear Quadratic Optimal Control Problem: A Reinforcement Learning Methodcited by 11
  4. 2022Logarithmic regret bounds for continuous-time average-reward Markov decision processescited by 8
  5. 2022Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimalitycited by 7
  6. 2022Square-root regret bounds for continuous-time episodic Markov decision processescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing
  2. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applications
  3. 2024Deep Reinforcement Learning for Online Optimal Execution Strategies
  4. 2026Mathematical methods of reinforcement learning
  5. 2025Optimizing Return Distributions with Distributional Dynamic Programming
  6. 2024Learning to Optimally Stop Diffusion Processes, with Financial Applications
  7. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  8. 2026Reinforcement Learning in Real Option Models