# Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes

arXiv: https://arxiv.org/abs/2006.11441 · HTML: https://mlsanity.com/paper/2006.11441/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 23 of 23)

- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [Continuous Coordination As a Realistic Scenario for Lifelong Learning](https://mlsanity.com/paper/2103.03216/) (2021)
- [Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training](https://mlsanity.com/paper/2202.09514/) (2022)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Building a Subspace of Policies for Scalable Continual Learning](https://mlsanity.com/paper/2211.10445/) (2022)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges](https://mlsanity.com/paper/2205.14495/) (2022)
- [Continual Vision-based Reinforcement Learning with Group Symmetries](https://mlsanity.com/paper/2210.12301/) (2022)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [Functional optimal transport: map estimation and domain adaptation for functional data](https://mlsanity.com/paper/2102.03895/) (2021)
- [Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables](https://mlsanity.com/paper/2210.12262/) (2022)
- [Online Reinforcement Learning in Non-Stationary Context-Driven Environments](https://mlsanity.com/paper/2302.02182/) (2023)
- [Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2303.08690/) (2023)
- [Linear Mixture Distributionally Robust Markov Decision Processes](https://mlsanity.com/paper/2505.18044/) (2025)
- [Global Convolutional Neural Processes](https://mlsanity.com/paper/2109.00691/) (2021)
- [On the Design of Safe Continual RL Methods for Control of Nonlinear Systems](https://mlsanity.com/paper/2502.15922/) (2025)
- [Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback](https://mlsanity.com/paper/2603.04029/) (2026)
- [Modeling Human Behavior Part I -- Learning and Belief Approaches](https://mlsanity.com/paper/2205.06485/) (2022)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Robustifying Reinforcement Learning Policies with L1 Adaptive Control](https://mlsanity.com/paper/2106.02249/) (2021)
- [Safe Continual Reinforcement Learning in Non-stationary Environments](https://mlsanity.com/paper/2604.19737/) (2026)

## Abstract (arXiv)

Continuously learning to solve unseen tasks with limited experience has been extensively pursued in meta-learning and continual learning, but with restricted assumptions such as accessible task distributions, independently and identically distributed tasks, and clear task delineations. However, real-world physical tasks frequently violate these assumptions, resulting in performance degradation. This paper proposes a continual online model-based reinforcement learning approach that does not require pre-training to solve task-agnostic problems with unknown task boundaries. We maintain a mixture of experts to handle nonstationarity, and represent each different type of dynamics with a Gaussian Process to efficiently leverage collected data and expressively model uncertainty. We propose a transition prior to account for the temporal dependencies in streaming data and update the mixture online via sequential variational inference. Our approach reliably handles the task distribution shift by generating new models for never-before-seen dynamics and reusing old models for previously seen dynamics. In experiments, our approach outperforms alternative methods in non-stationary tasks, including classic control with changing dynamics and decision making in different driving scenarios.