# DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2610.04494 · HTML: https://mlsanity.com/paper/2610.04494/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 3 Oct 2026.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 30 of 32)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [The Value Equivalence Principle for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2011.03506/) (2020)
- [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021)
- [Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components](https://mlsanity.com/paper/1804.06760/) (2018)
- [Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems](https://mlsanity.com/paper/2210.15432/) (2022)
- [Search-Based Testing of Reinforcement Learning](https://mlsanity.com/paper/2205.04887/) (2022)
- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [AmbieGen: A Search-based Framework for Autonomous Systems Testing](https://mlsanity.com/paper/2301.01234/) (2023)
- [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023)
- [Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification](https://mlsanity.com/paper/1910.02837/) (2019)
- [Pretrain, Prompt, and Transfer: Evolving Digital Twins for Time-to-Event Analysis in Cyber-physical Systems](https://mlsanity.com/paper/2310.00032/) (2023)
- [Policy Testing with MDPFuzz (Replicability Study)](https://mlsanity.com/paper/2502.19116/) (2025)
- [Knowledge Distillation-Empowered Digital Twin for Anomaly Detection](https://mlsanity.com/paper/2309.04616/) (2023)
- [Testing for Fault Diversity in Reinforcement Learning](https://mlsanity.com/paper/2403.15065/) (2024)
- [LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs](https://mlsanity.com/paper/2406.07467/) (2024)

## Abstract (arXiv)

Testing deep reinforcement learning (DRL) agents in cyber-physical systems aims to uncover diverse failures before deployment, but each execution can be expensive. Surrogate-assisted testing reduces this cost by learning to predict which test configurations are likely to fail. Prior surrogates treat the system as a black box and predict pass or fail outcomes directly; we instead model how a test unfolds and estimate failure from an imagined episode. We introduce DreamTest, a world-model surrogate for testing DRL agents. DreamTest adapts a recurrent state-space model to learn agent behaviour and environment dynamics from the agent's training log. Given a candidate configuration, imagined rollouts produce a failure score that guides search without executing every candidate in a simulator or real system. We evaluate DreamTest for failure prediction, test generation, and failure diversity on Parking, Humanoid, and DonkeyCar. Mean area under the precision-recall curve (AUPRC) exceeds the strongest baseline by 97%, 12%, and 39%, respectively, and gains on five out-of-distribution test sets reach 145%, 29%, and 44%. Under the same simulator-validation budget, the best "DreamTest + search" combinations find 29%, 22%, and 79% more novel failures on average. Across clusterings with k = 2-40, failures generated with DreamTest cover the most behavioural clusters for almost all k, indicating that DreamTest consistently discovers behaviourally diverse failures.