# On overfitting and asymptotic bias in batch reinforcement learning with partial observability

arXiv: https://arxiv.org/abs/1709.07796 · HTML: https://mlsanity.com/paper/1709.07796/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2017.

## Built on by (most cited first, 16 of 16)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Medical Dead-ends and Learning to Identify High-risk States and Treatments](https://mlsanity.com/paper/2110.04186/) (2021)
- [Semi-Markov Offline Reinforcement Learning for Healthcare](https://mlsanity.com/paper/2203.09365/) (2022)
- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [Agent-state based policies in POMDPs: Beyond belief-state MDPs](https://mlsanity.com/paper/2409.15703/) (2024)
- [Rethinking Value Function Learning for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2210.09960/) (2022)
- [Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access](https://mlsanity.com/paper/2509.26000/) (2025)
- [Model approximation in MDPs with unbounded per-step cost](https://mlsanity.com/paper/2402.08813/) (2024)
- [On shallow planning under partial observability](https://mlsanity.com/paper/2407.15820/) (2024)
- [A Covering Framework for Offline POMDPs Learning using Belief Space Metric](https://mlsanity.com/paper/2603.03191/) (2026)
- [POMRL: No-Regret Learning-to-Plan with Increasing Horizons](https://mlsanity.com/paper/2212.14530/) (2022)
- [Offline Risk-sensitive RL with Partial Observability to Enhance Performance in Human-Robot Teaming](https://mlsanity.com/paper/2402.05703/) (2024)
- [Attention on flow control: transformer-based reinforcement learning for lift regulation in highly disturbed flows](https://mlsanity.com/paper/2506.10153/) (2025)
- [Short-Term-to-Long-Term Memory Transfer for Knowledge Graphs under Partial Observability](https://mlsanity.com/paper/2605.22142/) (2026)
- [Neuro-Symbolic Meta-Policies for Temporal Knowledge-Graph Memory under Partial Observability](https://mlsanity.com/paper/2607.18368/) (2026)

## Abstract (arXiv)

This paper provides an analysis of the tradeoff between asymptotic bias (suboptimality with unlimited data) and overfitting (additional suboptimality due to limited data) in the context of reinforcement learning with partial observability. Our theoretical analysis formally characterizes that while potentially increasing the asymptotic bias, a smaller state representation decreases the risk of overfitting. This analysis relies on expressing the quality of a state representation by bounding L1 error terms of the associated belief states. Theoretical results are empirically illustrated when the state representation is a truncated history of observations, both on synthetic POMDPs and on a large-scale POMDP in the context of smartgrids, with real-world data. Finally, similarly to known results in the fully observable setting, we also briefly discuss and empirically illustrate how using function approximators and adapting the discount factor may enhance the tradeoff between asymptotic bias and overfitting in the partially observable context.