# Building surrogate models using trajectories of agents trained by Reinforcement Learning

arXiv: https://arxiv.org/abs/2509.01285 · HTML: https://mlsanity.com/paper/2509.01285/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 3 of its 31 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 3 of 3)

- [Explaining Deep Reinforcement Learning Agents In The Atari Domain through a Surrogate Model](https://mlsanity.com/paper/2110.03184/) (2021)
- [A spectral surrogate model for stochastic simulators computed from trajectory samples](https://mlsanity.com/paper/2207.05653/) (2022)
- [Model-Free Voltage Regulation of Unbalanced Distribution Network Based on Surrogate Model and Deep Reinforcement Learning](https://mlsanity.com/paper/2006.13992/) (2020)

## Built on by (most cited first, 1 of 1)

- [Optimizing Energy Management of Smart Grid using Reinforcement Learning aided by Surrogate models built using Physics-informed Neural Networks](https://mlsanity.com/paper/2510.17380/) (2025)

## Abstract (arXiv)

Sample efficiency in the face of computationally expensive simulations is a common concern in surrogate modeling. Current strategies to minimize the number of samples needed are not as effective in simulated environments with wide state spaces. As a response to this challenge, we propose a novel method to efficiently sample simulated deterministic environments by using policies trained by Reinforcement Learning. We provide an extensive analysis of these surrogate-building strategies with respect to Latin-Hypercube sampling or Active Learning and Kriging, cross-validating performances with all sampled datasets. The analysis shows that a mixed dataset that includes samples acquired by random agents, expert agents, and agents trained to explore the regions of maximum entropy of the state transition distribution provides the best scores through all datasets, which is crucial for a meaningful state space representation. We conclude that the proposed method improves the state-of-the-art and clears the path to enable the application of surrogate-aided Reinforcement Learning policy optimization strategies on complex simulators.