mlsanityData to 7 Oct 2026

A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning

arXiv 2206.04551 ↗First posted 9 Jun 2022cs.LG cs.AI cs.ROcited by 15

Not on the map. Why?

Abstract (arXiv)

The generalization of model-based reinforcement learning (MBRL) methods to environments with unseen transition dynamics is an important yet challenging problem. Existing methods try to extract environment-specified information Z from past transition segments to make the dynamics prediction model generalizable to different dynamics. However, because environments are not labelled, the extracted information inevitably contains redundant information unrelated to the dynamics in transition segments and thus fails to maintain a crucial property of Z: Z should be similar in the same environment and dissimilar in different ones. As a result, the learned dynamics prediction function will deviate from the true one, which undermines the generalization ability. To tackle this problem, we introduce an interventional prediction module to estimate the probability of two estimated hatzi, hatzj belonging to the same environment. Furthermore, by utilizing the Z's invariance within a single environment, a relational head is proposed to enforce the similarity between hatZ from the same environment. As a result, the redundant information will be reduced in hatZ. We empirically show that hatZ estimated by our method enjoy less redundant information than previous methods, and such hatZ can significantly reduce dynamics prediction errors and improve the performance of model-based RL methods on zero-shot new environments with unseen dynamics. The codes of this method are available at urlhttps://github.com/CR-Gjx/RIA.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

50 of 79 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  12. 2019Model-Based Reinforcement Learning for Ataricited by 495
38 more
  1. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  2. 2017A simple neural network module for relational reasoningcited by 345
  3. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  4. 2017Relation Networks for Object Detectioncited by 305
  5. 2019Causal Confusion in Imitation Learningcited by 286
  6. 2013Direct and Indirect Effectscited by 276
  7. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  8. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  9. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  10. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  11. 2019When to use parametric models in reinforcement learning?cited by 118
  12. 2020Invariant Causal Prediction for Block MDPscited by 104
  13. 2016Causal Bandits: Learning Good Interventions via Causal Inferencecited by 100
  14. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  15. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  16. 2018Confounding-Robust Policy Improvementcited by 73
  17. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  18. 2018Relational recurrent neural networkscited by 65
  19. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  20. 2019Data Efficient Reinforcement Learning for Legged Robotscited by 60
  21. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 57
  22. 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 46
  23. 2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
  24. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  25. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  26. 2019Environment Probing Interaction Policiescited by 39
  27. 2020Self-Supervised Relational Reasoning for Representation Learningcited by 37
  28. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  29. 2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 34
  30. 2019Learning Causal State Representations of Partially Observable Environmentscited by 30
  31. 2020Neural Methods for Point-wise Dependency Estimationcited by 22
  32. 2019Single Episode Policy Transfer in Reinforcement Learningcited by 17
  33. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  34. 2020Resolving Spurious Correlations in Causal Models of Environments via Interventionscited by 14
  35. 2017Discovering objects and their relations from entangled scene representationscited by 11
  36. 2020Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domainscited by 11
  37. 2020Intervention Design for Effective Sim2Real Transfercited by 9
  38. 2018Precision medicine as a control problem: Using simulation and deep reinforcement learning to discover adaptive, personalized multi-cytokine therapy for sepsiscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 7; 2024: 5.

15 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 43
  2. 2023Causal Reinforcement Learning: A Surveycited by 22
  3. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  4. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  5. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
  6. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  7. 2023Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learningcited by 8
  8. 2023ODE-based Recurrent Model-free Reinforcement Learning for POMDPscited by 6
  9. 2022Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learningcited by 3
  10. 2024What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learningcited by 3
  11. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  12. 2023Causal State Distillation for Explainable Reinforcement Learningcited by 2
3 more
  1. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learningcited by 2
  2. 2022Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learningcited by 0
  3. 2024Curricula for Learning Robust Policies with Factored State Representations in Changing Environmentscited by 0