# On the Generalization Gap in Reparameterizable Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.12654 · HTML: https://mlsanity.com/paper/1905.12654/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 22 of 22)

- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies](https://mlsanity.com/paper/2106.09678/) (2021)
- [When Is Generalizable Reinforcement Learning Tractable?](https://mlsanity.com/paper/2101.00300/) (2021)
- [Promoting Generalization for Exact Solvers via Adversarial Instance Augmentation](https://mlsanity.com/paper/2310.14161/) (2023)
- [RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation](https://mlsanity.com/paper/2602.01789/) (2026)
- [Model-free Policy Learning with Reward Gradients](https://mlsanity.com/paper/2103.05147/) (2021)
- [Offline Policy Optimization with Eligible Actions](https://mlsanity.com/paper/2207.00632/) (2022)
- [On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness](https://mlsanity.com/paper/2210.10464/) (2022)
- [Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomy](https://mlsanity.com/paper/2312.09436/) (2023)
- [Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence](https://mlsanity.com/paper/2402.02701/) (2024)
- [Robustness and Generalization in Quantum Reinforcement Learning via Lipschitz Regularization](https://mlsanity.com/paper/2410.21117/) (2024)
- [Provable Zero-Shot Generalization in Offline Reinforcement Learning](https://mlsanity.com/paper/2503.07988/) (2025)
- [Adversarial Style Transfer for Robust Policy Optimization in Deep Reinforcement Learning](https://mlsanity.com/paper/2308.15550/) (2023)
- [Efficient Training of Generalizable Visuomotor Policies via Control-Aware Augmentation](https://mlsanity.com/paper/2401.09258/) (2024)
- [Adapting Image-based RL Policies via Predicted Rewards](https://mlsanity.com/paper/2407.16842/) (2024)
- [Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments](https://mlsanity.com/paper/2506.21129/) (2025)
- [Sample-Efficient Neurosymbolic Deep Reinforcement Learning](https://mlsanity.com/paper/2601.02850/) (2026)
- [PRISM: Parallel Reward Integration with Symmetry for MORL](https://mlsanity.com/paper/2602.18277/) (2026)
- [Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2607.14506/) (2026)
- [DRIFT: Data Reduction via Informative Feature Transformation- Generalization Begins Before Deep Learning starts](https://mlsanity.com/paper/2506.19734/) (2025)
- [Rationality Measurement and Theory for Reinforcement Learning Agents](https://mlsanity.com/paper/2602.04737/) (2026)
- [Sample-efficient Neuro-symbolic Proximal Policy Optimization](https://mlsanity.com/paper/2604.25534/) (2026)

## Abstract (arXiv)

Understanding generalization in reinforcement learning (RL) is a significant challenge, as many common assumptions of traditional supervised learning theory do not apply. We focus on the special class of reparameterizable RL problems, where the trajectory distribution can be decomposed using the reparametrization trick. For this problem class, estimating the expected return is efficient and the trajectory can be computed deterministically given peripheral random variables, which enables us to study reparametrizable RL using supervised learning and transfer learning theory. Through these relationships, we derive guarantees on the gap between the expected and empirical return for both intrinsic and external errors, based on Rademacher complexity as well as the PAC-Bayes bound. Our bound suggests the generalization capability of reparameterizable RL is related to multiple factors including "smoothness" of the environment transition, reward and agent policy function class. We also empirically verify the relationship between the generalization gap and these factors through simulations.