Data to 14 Oct 2026

Block Contextual MDPs for Continual Learning

arXiv 2110.06972 ↗First posted 13 Oct 2021cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), when defining a Markov Decision Process (MDP), the environment dynamics is implicitly assumed to be stationary. This assumption of stationarity, while simplifying, can be unrealistic in many scenarios. In the continual reinforcement learning scenario, the sequence of tasks is another source of nonstationarity. In this work, we propose to examine this continual reinforcement learning setting through the block contextual MDP (BC-MDP) framework, which enables us to relax the assumption of stationarity. This framework challenges RL algorithms to handle both nonstationarity and rich observation settings and, by additionally leveraging smoothness properties, enables us to study generalization bounds for this setting. Finally, we take inspiration from adaptive control to propose a novel algorithm that addresses the challenges introduced by this more realistic BC-MDP setting, allows for zero-shot adaptation at evaluation time, and achieves strong performance on several nonstationary environments.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 52 of its 103 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

52 of 103 references.

References · 52

2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
2017Gradient Episodic Memory for Continual Learningcited by 1,898
2016Progressive Neural Networkscited by 1,361
2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
2016HyperNetworkscited by 1,027
2018DeepMind Control Suitecited by 956
2015Policy Distillationcited by 440
2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
2016Successor Features for Transfer in Reinforcement Learningcited by 340

38 more

2015Norm-Based Capacity Control in Neural Networkscited by 299
2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
2017Distral: Robust Multitask Reinforcement Learningcited by 233
2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
2015Contextual Markov Decision Processescited by 220
2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
2019Meta-Learning Representations for Continual Learningcited by 189
2018Reinforced Continual Learningcited by 186
2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 127
2018Deep Variational Reinforcement Learning for POMDPscited by 125
2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 98
2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 81
2019Meta reinforcement learning as task inferencecited by 77
2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
2019Hypernetwork functional image representationcited by 65
2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 63
2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
2017Markov Decision Processes with Continuous Side Informationcited by 56
2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
2018Hardware Conditioned Policies for Multi-Robot Transfer Learningcited by 46
2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 43
2019Environment Probing Interaction Policiescited by 39
2017Robust Locally-Linear Controllable Embeddingcited by 38
2018Continual Reinforcement Learning with Complex Synapsescited by 32
2019Learning Causal State Representations of Partially Observable Environmentscited by 30
2020Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trialscited by 20
2019Combining Physical Simulators and Object-Based Networks for Controlcited by 18
2020Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularizationcited by 16

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 6; 2024: 5; 2025: 6; 2026: 1.

25 papers.

Sort:

By area:

2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20
2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 11
2022Meta-Learning Parameterized Skillscited by 7

13 more

2023Provably Efficient CVaR RL in Low-rank MDPscited by 6
2024Sample Complexity Characterization for Linear Contextual MDPscited by 5
2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learningcited by 5
2025Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalizationcited by 5
2023Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representationcited by 4
2024Learning to Look: Seeking Information for Decision Making via Policy Factorizationcited by 3
2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
2025Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networkscited by 1
2026Perception-Based Beliefs for POMDPs with Visual Observationscited by 1
2024Zero-shot Generalization in Inventory Management: Train, then Estimate and Decidecited by 0
2025Wavelet Predictive Representations for Non-Stationary Reinforcement Learningcited by 0
2025Representative Action Selection for Large Action Space: From Bandits to MDPscited by 0
2025Poincar'e Meets Bellman: Revisable Memory, Operational Quotients, and Evidence-Supported Learning in Changing Environmentscited by 0