# From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models

arXiv: https://arxiv.org/abs/2507.18229 · HTML: https://mlsanity.com/paper/2507.18229/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 9 of 9)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics](https://mlsanity.com/paper/2004.01509/) (2020)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Reinforcement Learning for Mean Field Game](https://mlsanity.com/paper/1905.13357/) (2019)
- [Simulating the Economic Impact of Rationality through Reinforcement Learning and Agent-Based Modelling](https://mlsanity.com/paper/2405.02161/) (2024)
- [Analyzing Micro-Founded General Equilibrium Models with Many Agents using Deep Reinforcement Learning](https://mlsanity.com/paper/2201.01163/) (2022)
- [Empirical Equilibria in Agent-based Economic systems with Learning agents](https://mlsanity.com/paper/2408.12038/) (2024)

## Abstract (arXiv)

The application of Reinforcement Learning (RL) to economic modeling reveals a fundamental conflict between the assumptions of equilibrium theory and the emergent behavior of learning agents. While canonical economic models assume atomistic agents act as `takers' of aggregate market conditions, a naive single-agent RL simulation incentivizes the agent to become a `manipulator' of its environment. This paper first demonstrates this discrepancy within a search-and-matching model with concave production, showing that a standard RL agent learns a non-equilibrium, monopsonistic policy. Additionally, we identify a parametric bias arising from the mismatch between economic discounting and RL's treatment of intertemporal costs. To address both issues, we propose a calibrated Mean-Field Reinforcement Learning framework that embeds a representative agent in a fixed macroeconomic field and adjusts the cost function to reflect economic opportunity costs. Our iterative algorithm converges to a self-consistent fixed point where the agent's policy aligns with the competitive equilibrium. This approach provides a tractable and theoretically sound methodology for modeling learning agents in economic systems within the broader domain of computational social science.