Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 44 references that are in the map. First posted on arXiv 10 Oct 2026.
Builds on
23 of 44 references.
References · 23
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
- 2017Hindsight Experience Replaycited by 1,136
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
- 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
- 2019Learning Latent Plans from Playcited by 316
- 2019Causal Confusion in Imitation Learningcited by 288
- 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
- 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
11 more
- 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
- 2018Confounding-Robust Policy Improvementcited by 76
- 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
- 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
- 2022Causal Imitation Learning with Unobserved Confounderscited by 41
- 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 24
- 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 23
- 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
- 2024Consistency of Neural Causal Partial Identificationcited by 5
- 2024Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneitycited by 2
- 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
- 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
- 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
- 2024Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning
- 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
- 2025Model-Based Reinforcement Learning Under Confounding
- 2025Counterfactual experience augmented off-policy reinforcement learning
- 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning