mlsanityData to 7 Oct 2026

Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning

arXiv 2207.09081 ↗First posted 19 Jul 2022cs.LG cs.AI cs.ROcited by 33

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

As a pivotal component to attaining generalizable solutions in human intelligence, reasoning provides great potential for reinforcement learning (RL) agents' generalization towards varied goals by summarizing part-to-whole arguments and discovering cause-and-effect relations. However, how to discover and represent causalities remains a huge gap that hinders the development of causal RL. In this paper, we augment Goal-Conditioned RL (GCRL) with Causal Graph (CG), a structure built upon the relation between objects and events. We novelly formulate the GCRL problem into variational likelihood maximization with CG as latent variables. To optimize the derived objective, we propose a framework with theoretical performance guarantees that alternates between two steps: using interventional data to estimate the posterior of CG; using CG to learn generalizable models and interpretable policies. Due to the lack of public benchmarks that verify generalization capability under reasoning, we design nine tasks and then empirically show the effectiveness of the proposed method against five baselines on these tasks. Further theoretical analysis shows that our performance improvement is attributed to the virtuous cycle of causal discovery, transition modeling, and policy training, which aligns with the experimental evidence in extensive ablation studies.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 60 of its 107 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

60 of 107 references.

References · 60

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  3. 2019Invariant Risk Minimizationcited by 1,832
  4. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  11. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  12. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 335
38 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 318
  2. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  3. 2020Differentiable Causal Discovery from Interventional Datacited by 218
  4. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  5. 2019Active Domain Randomizationcited by 161
  6. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  7. 2019Causal Discovery with Reinforcement Learningcited by 145
  8. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  9. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  10. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  11. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  12. 2020Invariant Causal Prediction for Block MDPscited by 104
  13. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  14. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  15. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  16. 2018Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Datacited by 94
  17. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  18. 2020Causal Discovery in Physical Systems from Videoscited by 77
  19. 2018Characterizing and Learning Equivalence Classes of Causal DAGs under Interventionscited by 76
  20. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  21. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
  22. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  23. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 60
  24. 2020Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoningcited by 58
  25. 2019Exploration via Hindsight Goal Generationcited by 53
  26. 2020Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learningcited by 53
  27. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  28. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  29. 2016Towards Deep Symbolic Reinforcement Learningcited by 42
  30. 2021Learning Neural Causal Models with Active Interventionscited by 40
  31. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
  32. 2019Regression Planning Networkscited by 32
  33. 2019Causal Induction from Visual Observations for Goal Directed Taskscited by 31
  34. 2022A Theory of Abstraction in Reinforcement Learningcited by 26
  35. 2018Experimental Design for Cost-Aware Learning of Causal Graphscited by 25
  36. 2017Testing Conditional Independence of Discrete Distributionscited by 22
  37. 2020Efficient Intervention Design for Causal Discovery with Latentscited by 22
  38. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 10; 2024: 15; 2025: 7.

33 papers.

Sort:

By area:

  1. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  2. 2023A Survey on Causal Reinforcement Learningcited by 43
  3. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  4. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  5. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  6. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  7. 2023Causal Reinforcement Learning: A Surveycited by 22
  8. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  9. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  10. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 16
  11. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  12. 2024Causal Action Influence Aware Counterfactual Data Augmentationcited by 12
21 more
  1. 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learningcited by 11
  2. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  3. 2023Learning to View: Decision Transformers for Active Object Detectioncited by 10
  4. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 10
  5. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  6. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 4
  7. 2025Curious Causality-Seeking Agents Learn Meta Causal Worldcited by 4
  8. 2024Automated Discovery of Functional Actual Causes in Complex Environmentscited by 3
  9. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  10. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  11. 2023Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Modelscited by 2
  12. 2024Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learningcited by 2
  13. 2024Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representationscited by 2
  14. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1
  15. 2024TrafficGamer: Reliable and Flexible Traffic Simulation for Safety-Critical Scenarios with Game-Theoretic Oraclescited by 1
  16. 2025D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detectioncited by 1
  17. 2024Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtaskscited by 0
  18. 2025Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challengescited by 0
  19. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventionscited by 0
  20. 2025Incorporating Interventional Independence Improves Robustness against Interventional Distribution Shiftcited by 0
  21. 2025Goal Discovery with Causal Capacity for Efficient Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalization of RLVR Using Causal Reasoning as a Testbed
  2. 2025A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inference
  3. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning
  4. 2026An Introduction to Causal Reinforcement Learning
  5. 2025Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learning
  6. 2025Causal Process Models: Reframing Dynamic Causal Graph Discovery as a Reinforcement Learning Problem
  7. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning
  8. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation