# Lifelong Incremental Reinforcement Learning with Online Bayesian Inference

arXiv: https://arxiv.org/abs/2007.14196 · HTML: https://mlsanity.com/paper/2007.14196/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2020.

## Built on by (most cited first, 23 of 23)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Large Sequence Models for Sequential Decision-Making: A Survey](https://mlsanity.com/paper/2306.13945/) (2023)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement](https://mlsanity.com/paper/2410.11448/) (2024)
- [A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning](https://mlsanity.com/paper/2205.10787/) (2022)
- [Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization](https://mlsanity.com/paper/2209.00347/) (2022)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reduction](https://mlsanity.com/paper/2104.07282/) (2021)
- [Mixture-of-Experts Meets In-Context Reinforcement Learning](https://mlsanity.com/paper/2506.05426/) (2025)
- [Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision](https://mlsanity.com/paper/2504.15046/) (2025)
- [Scalable In-Context Q-Learning](https://mlsanity.com/paper/2506.01299/) (2025)
- [Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning](https://mlsanity.com/paper/2408.00309/) (2024)
- [Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07729/) (2022)
- [Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2307.08033/) (2023)
- [Incremental Online Learning of Randomized Neural Network with Forward Regularization](https://mlsanity.com/paper/2412.13096/) (2024)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Depthwise Convolution for Multi-Agent Communication with Enhanced Mean-Field Approximation](https://mlsanity.com/paper/2203.02896/) (2022)
- [Goal-oriented inference of environment from redundant observations](https://mlsanity.com/paper/2305.04432/) (2023)
- [BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization](https://mlsanity.com/paper/2308.01207/) (2023)
- [Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments](https://mlsanity.com/paper/2512.18670/) (2025)
- [Learning Transferable Skills in Action RPGs via Directed Skill Graphs and Selective Adaptation](https://mlsanity.com/paper/2601.17923/) (2026)

## Abstract (arXiv)

A central capability of a long-lived reinforcement learning (RL) agent is to incrementally adapt its behavior as its environment changes, and to incrementally build upon previous experiences to facilitate future learning in real-world scenarios. In this paper, we propose LifeLong Incremental Reinforcement Learning (LLIRL), a new incremental algorithm for efficient lifelong adaptation to dynamic environments. We develop and maintain a library that contains an infinite mixture of parameterized environment models, which is equivalent to clustering environment parameters in a latent space. The prior distribution over the mixture is formulated as a Chinese restaurant process (CRP), which incrementally instantiates new environment models without any external information to signal environmental changes in advance. During lifelong learning, we employ the expectation maximization (EM) algorithm with online Bayesian inference to update the mixture in a fully incremental manner. In EM, the E-step involves estimating the posterior expectation of environment-to-cluster assignments, while the M-step updates the environment parameters for future learning. This method allows for all environment models to be adapted as necessary, with new models instantiated for environmental changes and old models retrieved when previously seen environments are encountered again. Experiments demonstrate that LLIRL outperforms relevant existing methods, and enables effective incremental adaptation to various dynamic environments for lifelong learning.