mlsanityData to 14 Oct 2026

Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

arXiv 2512.18892 ↗First posted 21 Dec 2025econ.TH cs.AI cs.LGcited by 2

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 76 references that are in the map. First posted on arXiv 21 Dec 2025.

Builds on

12 of 76 references.

References · 12

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  4. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  5. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  6. 2022Learning in Mean Field Games: A Surveycited by 61
  7. 2021Deep Reinforcement Learning in a Monetary Modelcited by 11
  8. 2021DeepHAM: A Global Solution Method for Heterogeneous Agent Models with Aggregate Shockscited by 11
  9. 2023Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learningcited by 4
  10. 2025Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learningcited by 4
  11. 2024Global Solutions to Master Equations for Continuous Time Heterogeneous Agent Macroeconomic Modelscited by 2
  12. 2025Heterogeneous RBCs via Deep Multi-Agent Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Recurrent Structural Policy Gradient for Partially Observable Mean Field Gamescited by 0
  2. 2026Beyond Nominal Equilibria: Risk-Averse Multi-Population Mean-Field Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models
  2. 2024Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning
  3. 2024Empirical Equilibria in Agent-based Economic systems with Learning agents
  4. 2026A Survey of Reinforcement Learning For Economics
  5. 2024Simulating the Economic Impact of Rationality through Reinforcement Learning and Agent-Based Modelling
  6. 2024The Evolution of Reinforcement Learning in Quantitative Finance: A Survey
  7. 2024Learning Macroeconomic Policies through Dynamic Stackelberg Mean-Field Games
  8. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities