mlsanityData to 14 Oct 2026

Epistemically-guided forward-backward exploration

arXiv 2507.05477 ↗First posted 7 Jul 2025cs.LG cs.AIcited by 8

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Zero-shot reinforcement learning is necessary for extracting optimal policies in absence of concrete rewards for fast adaptation to future problem settings. Forward-backward representations (FB) have emerged as a promising method for learning optimal policies in absence of rewards via a factorization of the policy occupancy measure. However, up until now, FB and many similar zero-shot reinforcement learning algorithms have been decoupled from the exploration problem, generally relying on other exploration algorithms for data collection. We argue that FB representations should fundamentally be used for exploration in order to learn more efficiently. With this goal in mind, we design exploration policies that arise naturally from the FB representation that minimize the posterior variance of the FB representation, hence minimizing its epistemic uncertainty. We empirically demonstrate that such principled exploration strategies improve sample complexity of the FB algorithm considerably in comparison to other exploration methods. Code is publicly available at https://sites.google.com/view/fbee-url.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 56 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

35 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models15 Apr
  2. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization16 Dec

Other references · 33

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2018DeepMind Control Suitecited by 956
  5. 2017Deep Bayesian Active Learning with Image Datacited by 819
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2016Deep Exploration via Bootstrapped DQNcited by 527
  8. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  9. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  10. 2019Self-Supervised Exploration via Disagreementcited by 265
  11. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  12. 2019Efficient Exploration via State Marginal Matchingcited by 188
21 more
  1. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  2. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  3. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  4. 2021Discovering and Achieving Goals via World Modelscited by 131
  5. 2021Learning One Representation to Optimize All Rewardscited by 115
  6. 2021APS: Active Pretraining with Successor Featurescited by 112
  7. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  8. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  9. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  10. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  11. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  12. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  13. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  14. 2021Learning more skills through optimistic explorationcited by 38
  15. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  16. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 31
  17. 2024Transductive Active Learning: Theory and Applicationscited by 22
  18. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  19. 2024Language Guided Skill Discoverycited by 16
  20. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  21. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 22
  2. 2025Optimistic Task Inference for Behavior Foundation Modelscited by 6
  3. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 3
  4. 2026Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulationcited by 3
  5. 2025A Unified Framework for Zero-Shot Reinforcement Learningcited by 1
  6. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learningcited by 1
  7. 2026Spectral Alignment in Forward-Backward Representations via Temporal Abstractioncited by 0
  8. 2026Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can We Really Learn One Representation to Optimize All Rewards?
  2. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weighting
  3. 2025On Efficient Bayesian Exploration in Model-Based Reinforcement Learning
  4. 2025Towards Robust Zero-Shot Reinforcement Learning
  5. 2025Exploration with Foundation Models: Capabilities, Limitations, and Hybrid Approaches
  6. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models
  7. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning
  8. 2024Zero-Shot Reinforcement Learning via Function Encoders