# Reinforcement Learning for Market Making in a Multi-agent Dealer Market

arXiv: https://arxiv.org/abs/1911.05892 · HTML: https://mlsanity.com/paper/1911.05892/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2019.

## Built on by (most cited first, 30 of 39)

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance](https://mlsanity.com/paper/2111.09395/) (2021)
- [Transfer RL across Observation Feature Spaces via Model-Based Regularization](https://mlsanity.com/paper/2201.00248/) (2022)
- [JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading](https://mlsanity.com/paper/2308.13289/) (2023)
- [A Review of Reinforcement Learning in Financial Applications](https://mlsanity.com/paper/2411.12746/) (2024)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [Towards a fully RL-based Market Simulator](https://mlsanity.com/paper/2110.06829/) (2021)
- [Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations](https://mlsanity.com/paper/2210.07184/) (2022)
- [Market Making with Scaled Beta Policies](https://mlsanity.com/paper/2207.03352/) (2022)
- [Phantom -- A RL-driven multi-agent framework to model complex systems](https://mlsanity.com/paper/2210.06012/) (2022)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Tractable Equilibrium Computation in Markov Games through Risk Aversion](https://mlsanity.com/paper/2406.14156/) (2024)
- [Architecture of Automated Crypto-Finance Agent](https://mlsanity.com/paper/2107.07769/) (2021)
- [Model-based gym environments for limit order book trading](https://mlsanity.com/paper/2209.07823/) (2022)
- [Influence-Augmented Local Simulators: A Scalable Solution for Fast Deep RL in Large Networked Systems](https://mlsanity.com/paper/2202.01534/) (2022)
- [Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model](https://mlsanity.com/paper/2207.09951/) (2022)
- [Modelling Opaque Bilateral Market Dynamics in Financial Trading: Insights from a Multi-Agent Simulation Study](https://mlsanity.com/paper/2405.02849/) (2024)
- [Decoding OTC Government Bond Market Liquidity: An ABM Model for Market Dynamics](https://mlsanity.com/paper/2501.16331/) (2024)
- [Multi-Agent Reinforcement Learning for Market Making: Competition without Collusion](https://mlsanity.com/paper/2510.25929/) (2025)
- [Training Generalizable Collaborative Agents via Strategic Risk Aversion](https://mlsanity.com/paper/2602.21515/) (2026)
- [Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market](https://mlsanity.com/paper/2112.04494/) (2021)
- [Hedging and Pricing Structured Products Featuring Multiple Underlying Assets](https://mlsanity.com/paper/2411.01121/) (2024)
- [Think Smart, Act SMARL! Analyzing Probabilistic Logic Shields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2411.04867/) (2024)
- [EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2501.15129/) (2025)
- [Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics](https://mlsanity.com/paper/2509.12456/) (2025)
- [Adaptive Agents in Spatial Double-Auction Markets: Modeling the Emergence of Industrial Symbiosis](https://mlsanity.com/paper/2512.17979/) (2025)
- [Learning Market Making with Closing Auctions](https://mlsanity.com/paper/2601.17247/) (2026)

## Abstract (arXiv)

Market makers play an important role in providing liquidity to markets by continuously quoting prices at which they are willing to buy and sell, and managing inventory risk. In this paper, we build a multi-agent simulation of a dealer market and demonstrate that it can be used to understand the behavior of a reinforcement learning (RL) based market maker agent. We use the simulator to train an RL-based market maker agent with different competitive scenarios, reward formulations and market price trends (drifts). We show that the reinforcement learning agent is able to learn about its competitor's pricing policy; it also learns to manage inventory by smartly selecting asymmetric prices on the buy and sell sides (skewing), and maintaining a positive (or negative) inventory depending on whether the market price drift is positive (or negative). Finally, we propose and test reward formulations for creating risk averse RL-based market maker agents.