# Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations

arXiv: https://arxiv.org/abs/2210.07184 · HTML: https://mlsanity.com/paper/2210.07184/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 92 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [The Numerics of GANs](https://mlsanity.com/paper/1705.10461/) (2017)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games](https://mlsanity.com/paper/2106.01969/) (2021)
- [The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies](https://mlsanity.com/paper/2004.13332/) (2020)
- [Real World Games Look Like Spinning Tops](https://mlsanity.com/paper/2004.09468/) (2020)
- [Market Making via Reinforcement Learning](https://mlsanity.com/paper/1804.04216/) (2018)
- [Differentiable Game Mechanics](https://mlsanity.com/paper/1905.04926/) (2019)
- [Reinforcement Learning for Market Making in a Multi-agent Dealer Market](https://mlsanity.com/paper/1911.05892/) (2019)
- [Actor-Critic Policy Optimization in Partially Observable Multiagent Environments](https://mlsanity.com/paper/1810.09026/) (2018)
- [Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descent](https://mlsanity.com/paper/1903.05614/) (2019)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games](https://mlsanity.com/paper/2107.13090/) (2021)
- [Learning to Optimize in Swarms](https://mlsanity.com/paper/1911.03787/) (2019)
- [Calibration of Shared Equilibria in General Sum Partially Observable Markov Games](https://mlsanity.com/paper/2006.13085/) (2020)
- [Variance Adjusted Actor Critic Algorithms](https://mlsanity.com/paper/1310.3697/) (2013)

## Built on by (most cited first, 10 of 10)

- [Agentic AI Systems Applied to tasks in Financial Services: Modeling and model risk management crews](https://mlsanity.com/paper/2502.05439/) (2025)
- [Reinforcement Learning for Combining Search Methods in the Calibration of Economic ABMs](https://mlsanity.com/paper/2302.11835/) (2023)
- [Simulating the Economic Impact of Rationality through Reinforcement Learning and Agent-Based Modelling](https://mlsanity.com/paper/2405.02161/) (2024)
- [ADAGE: A generic two-layer framework for adaptive agent based modelling](https://mlsanity.com/paper/2501.09429/) (2025)
- [Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior](https://mlsanity.com/paper/2403.19781/) (2024)
- [Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.00787/) (2024)
- [Modelling Opaque Bilateral Market Dynamics in Financial Trading: Insights from a Multi-Agent Simulation Study](https://mlsanity.com/paper/2405.02849/) (2024)
- [Decoding OTC Government Bond Market Liquidity: An ABM Model for Market Dynamics](https://mlsanity.com/paper/2501.16331/) (2024)
- [Shifting Power: Leveraging LLMs to Simulate Human Aversion in ABMs of Bilateral Financial Exchanges, A bond market study](https://mlsanity.com/paper/2503.00320/) (2025)
- [Reinforcement Learning in High-frequency Market Making](https://mlsanity.com/paper/2407.21025/) (2024)

## Abstract (arXiv)

We study a game between liquidity provider and liquidity taker agents interacting in an over-the-counter market, for which the typical example is foreign exchange. We show how a suitable design of parameterized families of reward functions coupled with shared policy learning constitutes an efficient solution to this problem. By playing against each other, our deep-reinforcement-learning-driven agents learn emergent behaviors relative to a wide spectrum of objectives encompassing profit-and-loss, optimal execution and market share. In particular, we find that liquidity providers naturally learn to balance hedging and skewing, where skewing refers to setting their buy and sell prices asymmetrically as a function of their inventory. We further introduce a novel RL-based calibration algorithm which we found performed well at imposing constraints on the game equilibrium. On the theoretical side, we are able to show convergence rates for our multi-agent policy gradient algorithm under a transitivity assumption, closely related to generalized ordinal potential games.