mlsanityData to 7 Oct 2026

AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning

arXiv 2107.02729 ↗First posted 6 Jul 2021cs.LG cs.AI stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

One practical challenge in reinforcement learning (RL) is how to make quick adaptations when faced with new environments. In this paper, we propose a principled framework for adaptive RL, called textitAdaRL, that adapts reliably and efficiently to changes across domains with a few samples from the target domain, even in partially observable environments. Specifically, we leverage a parsimonious graphical representation that characterizes structural relationships over variables in the RL system. Such graphical representations provide a compact way to encode what and where the changes across domains are, and furthermore inform us with a minimal set of changes that one has to consider for the purpose of policy adaptation. We show that by explicitly leveraging this compact representation to encode changes, we can efficiently adapt the policy to the target domain, in which only a few samples are needed and further policy optimization is avoided. We illustrate the efficacy of AdaRL through a series of experiments that vary factors in the observation, transition, and reward functions for Cartpole and Atari games.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 39 of its 81 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

39 of 81 references.

References · 39

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016Progressive Neural Networkscited by 1,332
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
27 more
  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  3. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  4. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  5. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  6. 2019Causality for Machine Learningcited by 242
  7. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  8. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  9. 2021Towards Causal Representation Learningcited by 207
  10. 2019Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changescited by 166
  11. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  12. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  13. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  14. 2020Invariant Causal Prediction for Block MDPscited by 104
  15. 2014Optimizing the CVaR via Samplingcited by 98
  16. 2019Meta-Learning without Memorizationcited by 92
  17. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  18. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  19. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  20. 2019Meta-Q-Learningcited by 62
  21. 2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translationcited by 50
  22. 2018Unsupervised Meta-Learning for Reinforcement Learningcited by 38
  23. 2020Domain Adaptation as a Problem of Inference on Graphical Modelscited by 36
  24. 2019Guided Meta-Policy Searchcited by 26
  25. 2021Model-Invariant State Abstractions for Model-Based Reinforcement Learningcited by 18
  26. 2018Importance Weighted Transfer of Samples in Reinforcement Learningcited by 16
  27. 2018Bayesian Reinforcement Learning in Factored POMDPscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 16; 2024: 14; 2025: 4; 2026: 2.

46 papers.

Sort:

By area:

  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  2. 2022Temporally Disentangled Representation Learningcited by 59
  3. 2023A Survey on Causal Reinforcement Learningcited by 43
  4. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  5. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  6. 2023Causal Reinforcement Learning: A Surveycited by 22
  7. 2023Temporally Disentangled Representation Learning under Unknown Nonstationaritycited by 21
  8. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  9. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  10. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 17
  11. 2022Variational Causal Dynamics: Discovering Modular World Models from Interventionscited by 16
  12. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
34 more
  1. 2022Learning Mixtures of Linear Dynamical Systemscited by 15
  2. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  3. 2023Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learningcited by 15
  4. 2022Learning Latent Causal Dynamicscited by 13
  5. 2024Causal Temporal Representation Learning with Nonstationary Sparse Transitioncited by 12
  6. 2024SPARTAN: A Sparse Transformer World Model Attending to What Matterscited by 10
  7. 2023Efficient Symbolic Policy Learning with Differentiable Symbolic Expressioncited by 9
  8. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  9. 2023Exploiting Causal Graph Priors with Posterior Sampling for Reinforcement Learningcited by 5
  10. 2024A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Pointscited by 5
  11. 2022Sampling Through the Lens of Sequential Decision Makingcited by 4
  12. 2023Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representationcited by 4
  13. 2023Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representationscited by 4
  14. 2024Compete and Compose: Learning Independent Mechanisms for Modular World Modelscited by 4
  15. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 4
  16. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3
  17. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  18. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  19. 2025PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policiescited by 3
  20. 2025Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learningcited by 3
  21. 2023Reinforcement Learning with Temporal-Logic-Based Causal Diagramscited by 2
  22. 2024Pausing Policy Learning in Non-stationary Reinforcement Learningcited by 2
  23. 2024Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representationscited by 2
  24. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  25. 2023Heterogeneous Knowledge for Augmented Modular Reinforcement Learningcited by 1
  26. 2023Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learningcited by 1
  27. 2024Preparing for Black Swans: The Antifragility Imperative for Machine Learningcited by 1
  28. 2024New Rules for Causal Identification with Background Knowledgecited by 1
  29. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1
  30. 2024Stepping Out of the Shadows: Reinforcement Learning in Shadow Modecited by 1
  31. 2025Learning Causal Structure Distributions for Robust Planningcited by 1
  32. 2022A Survey on Causal Representation Learning and Future Work for Medical Image Analysiscited by 0
  33. 2024Identifying latent state transition in non-linear dynamical systemscited by 0
  34. 2026Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmarkcited by 0