mlsanityData to 14 Oct 2026

Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory

arXiv 2411.00401 ↗First posted 1 Nov 2024cs.LG cs.AIcited by 7

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Lifelong reinforcement learning (RL) has been developed as a paradigm for extending single-task RL to more realistic, dynamic settings. In lifelong RL, the "life" of an RL agent is modeled as a stream of tasks drawn from a task distribution. We propose EPIC (Empirical PAC-Bayes that Improves Continuously), a novel algorithm designed for lifelong RL using PAC-Bayes theory. EPIC learns a shared policy distribution, referred to as the world policy, which enables rapid adaptation to new tasks while retaining valuable knowledge from previous experiences. Our theoretical analysis establishes a relationship between the algorithm's generalization performance and the number of prior tasks preserved in memory. We also derive the sample complexity of EPIC in terms of RL regret. Extensive experiments on a variety of environments demonstrate that EPIC significantly outperforms existing methods in lifelong RL, offering both theoretical guarantees and practical efficacy through the use of the world policy.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 54 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

37 of 54 references.

References · 37

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018World Modelscited by 1,513
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  10. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  11. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
25 more
  1. 2023A Definition of Continual Reinforcement Learningcited by 101
  2. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  3. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  4. 2020learn2learn: A Library for Meta-Learning Researchcited by 69
  5. 2020On the role of data in PAC-Bayes boundscited by 64
  6. 2022A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learningcited by 31
  7. 2015Data-dependent Initializations of Convolutional Neural Networkscited by 29
  8. 2017Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processescited by 29
  9. 2020Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgettingcited by 29
  10. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  11. 2020Lipschitz Lifelong Reinforcement Learningcited by 23
  12. 2018PAC-Bayes Control: Learning Policies that Provably Generalize to Novel Environmentscited by 22
  13. 2024Efficient Frameworks for Generalized Low-Rank Matrix Bandit Problemscited by 20
  14. 2023Prediction and Control in Continual Reinforcement Learningcited by 19
  15. 2019Lifelong Learning with a Changing Action Setcited by 17
  16. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 17
  17. 2020Probably Approximately Correct Vision-Based Planning using Motion Primitivescited by 16
  18. 2023Statistical Guarantees for Variational Autoencoders using PAC-Bayesian Theorycited by 16
  19. 2023Robust Lipschitz Bandits to Adversarial Corruptionscited by 15
  20. 2022Model-based Lifelong Reinforcement Learning with Bayesian Explorationcited by 12
  21. 2021Syndicated Bandits: A Framework for Auto Tuning Hyper-parameters in Contextual Bandit Algorithmscited by 10
  22. 2018Planning and Learning with Stochastic Action Setscited by 9
  23. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9
  24. 2022PAC-Bayesian Lifelong Learning For Multi-Armed Banditscited by 4
  25. 2021Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularizationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 5.

7 papers.

  1. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  2. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 7
  3. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  4. 2024Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theorycited by 3
  5. 2025CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systemscited by 3
  6. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  7. 2025PAC-Bayesian Reinforcement Learning Trains Generalizable Policiescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning
  2. 2026Reinforcement Learning for Reachability: Guaranteeing Asymptotic Optimality
  3. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
  4. 2025Rethinking the Foundations for Continual Reinforcement Learning
  5. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  6. 2026Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specifications
  7. 2026DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guarantees
  8. 2026Maximum Likelihood Reinforcement Learning