# Online Planning in POMDPs with Self-Improving Simulators

arXiv: https://arxiv.org/abs/2201.11404 · HTML: https://mlsanity.com/paper/2201.11404/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Survey of multifidelity methods in uncertainty propagation, inference, and optimization](https://mlsanity.com/paper/1806.10761/) (2018)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Learning and Querying Fast Generative Models for Reinforcement Learning](https://mlsanity.com/paper/1802.03006/) (2018)
- [Learning Compact Models for Planning with Exogenous Processes](https://mlsanity.com/paper/1909.13870/) (2019)
- [A Sufficient Statistic for Influence in Structured Multiagent Environments](https://mlsanity.com/paper/1907.09278/) (2019)
- [Loss Bounds for Approximate Influence-Based Abstraction](https://mlsanity.com/paper/2011.01788/) (2020)
- [Influence-Augmented Online Planning for Complex Environments](https://mlsanity.com/paper/2010.11038/) (2020)

## Abstract (arXiv)

How can we plan efficiently in a large and complex environment when the time budget is limited? Given the original simulator of the environment, which may be computationally very demanding, we propose to learn online an approximate but much faster simulator that improves over time. To plan reliably and efficiently while the approximate simulator is learning, we develop a method that adaptively decides which simulator to use for every simulation, based on a statistic that measures the accuracy of the approximate simulator. This allows us to use the approximate simulator to replace the original simulator for faster simulations when it is accurate enough under the current context, thus trading off simulation speed and accuracy. Experimental results in two large domains show that when integrated with POMCP, our approach allows to plan with improving efficiency over time.