# Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.00787 · HTML: https://mlsanity.com/paper/2402.00787/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 71 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games](https://mlsanity.com/paper/1906.00729/) (2019)
- [Get Real: Realism Metrics for Robust Limit Order Book Market Simulations](https://mlsanity.com/paper/1912.04941/) (2019)
- [If MaxEnt RL is the Answer, What is the Question?](https://mlsanity.com/paper/1910.01913/) (2019)
- [Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning](https://mlsanity.com/paper/1901.09216/) (2019)
- [Calibration of Shared Equilibria in General Sum Partially Observable Markov Games](https://mlsanity.com/paper/2006.13085/) (2020)
- [Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations](https://mlsanity.com/paper/2210.07184/) (2022)
- [A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback](https://mlsanity.com/paper/2102.09180/) (2021)
- [Phantom -- A RL-driven multi-agent framework to model complex systems](https://mlsanity.com/paper/2210.06012/) (2022)
- [Reinforcement Learning for Economic Policy: A New Frontier?](https://mlsanity.com/paper/2206.08781/) (2022)
- [Roll-Drop: accounting for observation noise with a single parameter](https://mlsanity.com/paper/2304.13150/) (2023)
- [Bounded strategic reasoning explains crisis emergence in multi-agent market games](https://mlsanity.com/paper/2206.05568/) (2022)

## Built on by (most cited first, 2 of 2)

- [Tractable Equilibrium Computation in Markov Games through Risk Aversion](https://mlsanity.com/paper/2406.14156/) (2024)
- [Modelling bounded rational decision-making through Wasserstein constraints](https://mlsanity.com/paper/2504.03743/) (2025)

## Abstract (arXiv)

Agent-based models (ABMs) have shown promise for modelling various real world phenomena incompatible with traditional equilibrium analysis. However, a critical concern is the manual definition of behavioural rules in ABMs. Recent developments in multi-agent reinforcement learning (MARL) offer a way to address this issue from an optimisation perspective, where agents strive to maximise their utility, eliminating the need for manual rule specification. This learning-focused approach aligns with established economic and financial models through the use of rational utility-maximising agents. However, this representation departs from the fundamental motivation for ABMs: that realistic dynamics emerging from bounded rationality and agent heterogeneity can be modelled. To resolve this apparent disparity between the two approaches, we propose a novel technique for representing heterogeneous processing-constrained agents within a MARL framework. The proposed approach treats agents as constrained optimisers with varying degrees of strategic skills, permitting departure from strict utility maximisation. Behaviour is learnt through repeated simulations with policy gradients to adjust action likelihoods. To allow efficient computation, we use parameterised shared policy learning with distributions of agent skill levels. Shared policy learning avoids the need for agents to learn individual policies yet still enables a spectrum of bounded rational behaviours. We validate our model's effectiveness using real-world data on a range of canonical n-agent settings, demonstrating significantly improved predictive capability.