mlsanityData to 14 Oct 2026

Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics

arXiv 2509.12456 ↗First posted 15 Sep 2025q-fin.TR cs.AIcited by 1

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent's performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 27 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

8 of 27 references.

References · 8

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2019Reinforcement Learning for Market Making in a Multi-agent Dealer Marketcited by 39
  5. 2023JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for tradingcited by 14
  6. 2018Optimizing Market Making using Multi-Agent Reinforcement Learningcited by 13
  7. 2021Optimal Market Making by Reinforcement Learningcited by 4
  8. 2023Integrating Tick-level Data and Periodical Signal for High-frequency Market Makingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ViperQ: Order Flow Pattern Recognition via Auction Market Theory for Reinforcement Learning Tradingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior
  2. 2024Optimal Execution with Reinforcement Learning
  3. 2025Right Place, Right Time: Market Simulation-based RL for Execution Optimisation
  4. 2026When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game
  5. 2025When AI Trading Agents Compete: Adverse Selection of Meta-Orders by Reinforcement Learning-Based Market Making
  6. 2026Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning
  7. 2019Towards Inverse Reinforcement Learning for Limit Order Book Dynamics
  8. 2025Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution