mlsanityData to 14 Oct 2026

Meta-Reinforcement Learning by Tracking Task Non-stationarity

arXiv 2105.08834 ↗First posted 18 May 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Many real-world domains are subject to a structured non-stationarity which affects the agent's goals and the environmental dynamics. Meta-reinforcement learning (RL) has been shown successful for training agents that quickly adapt to related tasks. However, most of the existing meta-RL algorithms for non-stationary domains either make strong assumptions on the task generation process or require sampling from it at training time. In this paper, we propose a novel algorithm (TRIO) that optimizes for the future by explicitly tracking the task evolution through time. At training time, TRIO learns a variational module to quickly identify latent parameters from experience samples. This module is learned jointly with an optimal exploration policy that takes task uncertainty into account. At test time, TRIO tracks the evolution of the latent parameters online, hence reducing the uncertainty over future tasks and obtaining fast adaptation through the meta-learned policy. Unlike most existing methods, TRIO does not assume Markovian task-evolution processes, it does not require information about the non-stationarity at training time, and it captures complex changes undergoing in the environment. We evaluate our algorithm on different simulated problems and show it outperforms competitive baselines.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 18 May 2021.

Builds on

21 of 31 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2016Variational Inference: A Review for Statisticianscited by 1,472
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  5. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  7. 2016Learning to reinforcement learncited by 377
  8. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  9. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  10. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  11. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  12. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 81
9 more
  1. 2019Meta reinforcement learning as task inferencecited by 77
  2. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  3. 2020A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environmentscited by 62
  4. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44
  5. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 43
  6. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  7. 2020Fast Online Adaptation in Robotics through Meta-Learning Embeddings of Simulated Priorscited by 22
  8. 2020Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularizationcited by 16
  9. 2019Meta Reinforcement Learning with Task Embedding and Shared Policycited by 15

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 0; 2025: 1.

8 papers.

  1. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  2. 2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
  3. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  4. 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularizationcited by 5
  5. 2023Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representationcited by 4
  6. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2
  7. 2023Successive Model-Agnostic Meta-Learning for Few-Shot Fault Time Series Prognosiscited by 0
  8. 2025Wavelet Predictive Representations for Non-Stationary Reinforcement Learningcited by 0