# An Optimal Tightness Bound for the Simulation Lemma

arXiv: https://arxiv.org/abs/2406.16249 · HTML: https://mlsanity.com/paper/2406.16249/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 7 of 7)

- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Lipschitz Continuity in Model-based Reinforcement Learning](https://mlsanity.com/paper/1804.07193/) (2018)
- [Near Optimal Behavior via Approximate State Abstraction](https://mlsanity.com/paper/1701.04113/) (2017)
- [Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning](https://mlsanity.com/paper/2001.10742/) (2020)
- [Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2306.03186/) (2023)
- [Combining Parametric and Nonparametric Models for Off-Policy Evaluation](https://mlsanity.com/paper/1905.05787/) (2019)
- [Adaptive Belief Discretization for POMDP Planning](https://mlsanity.com/paper/2104.07276/) (2021)

## Built on by (most cited first, 8 of 8)

- [Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks](https://mlsanity.com/paper/2503.01885/) (2025)
- [Imperfect World Models are Exploitable](https://mlsanity.com/paper/2605.15960/) (2026)
- [Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning](https://mlsanity.com/paper/2605.29032/) (2026)
- [Zero-shot Generalization in Inventory Management: Train, then Estimate and Decide](https://mlsanity.com/paper/2411.00515/) (2024)
- [On Training in Imagination](https://mlsanity.com/paper/2605.06732/) (2026)
- [Smaller Abstract State Spaces Enable Cross-Scale Generalization in Reinforcement Learning](https://mlsanity.com/paper/2605.20272/) (2026)
- [Mind the Sim-to-Real Gap & Think Like a Scientist](https://mlsanity.com/paper/2605.21458/) (2026)
- [Robust and Learned Online Matching in Growing Trees](https://mlsanity.com/paper/2609.40077/) (2026)

## Abstract (arXiv)

We present a bound for value-prediction error with respect to model misspecification that is tight, including constant factors. This is a direct improvement of the "simulation lemma," a foundational result in reinforcement learning. We demonstrate that existing bounds are quite loose, becoming vacuous for large discount factors, due to the suboptimal treatment of compounding probability errors. By carefully considering this quantity on its own, instead of as a subcomponent of value error, we derive a bound that is sub-linear with respect to transition function misspecification. We then demonstrate broader applicability of this technique, improving a similar bound in the related subfield of hierarchical abstraction.