mlsanityData to 14 Oct 2026

Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning

arXiv 2605.13207 ↗First posted 13 May 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Hierarchical reinforcement learning can improve generalization by decomposing long-horizon decision-making into simpler subproblems. However, existing approaches often rely on restrictive design choices, such as fixed temporal abstractions or goal-conditioned objectives, which largely confine them to goal-reaching tasks and limit their applicability to general reward functions. In this paper, we introduce switching successor measures, an extension of successor measures that enables hierarchical control in zero-shot reinforcement learning without additional supervision, fixed horizons, or manually designed subgoals. We show that switching successor measures arise naturally from classical successor measures while preserving their underlying structure. Building on this result, we propose FB π-Switch, an algorithm that extracts both a high-level subgoal-selection policy and a low-level control policy directly from forward-backward (FB) representations, allowing hierarchical behavior to emerge from a single learned representation. Experiments on both goal-conditioned and general reward-based tasks show that FB π-Switch improves over non-hierarchical baselines and matches state-of-the-art hierarchical methods in goal-conditioned settings. These results demonstrate that structured successor representations provide a flexible foundation for hierarchical zero-shot reinforcement learning beyond goal-reaching tasks. Our project website is available at: https://stestokth.github.io/switching-successors/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 52 of its 62 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

52 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Can We Really Learn One Representation to Optimize All Rewards?11 Feb
  2. 2025Towards Robust Zero-Shot Reinforcement Learning17 Oct
  3. 2025Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning5 Sep
  4. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning16 Jun
  5. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models15 Apr
  6. 2025Fast Adaptation with Behavioral Foundation Models10 Apr
  7. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities19 Mar
  8. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weighting5 Dec
  9. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agent29 Nov
  10. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making24 Jun
  11. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings27 Feb
  12. 2024Foundation Policies with Hilbert Representations23 Feb

Other references · 40

  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  7. 2016The Option-Critic Architecturecited by 448
  8. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  9. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  10. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  11. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  12. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
28 more
  1. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  2. 2021Learning One Representation to Optimize All Rewardscited by 115
  3. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  4. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  5. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  6. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  7. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  8. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  9. 2022Skill-based Model-based Reinforcement Learningcited by 50
  10. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  11. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  12. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  13. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 27
  14. 2019Successor Options: An Option Discovery Framework for Reinforcement Learningcited by 23
  15. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  16. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 23
  17. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 21
  18. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  19. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  20. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 13
  21. 2026Compositional Planning with Jumpy World Modelscited by 7
  22. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  23. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  24. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RLcited by 5
  25. 2017Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learningcited by 4
  26. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learningcited by 4
  27. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  28. 2024To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Distributional Successor Features Enable Zero-Shot Policy Optimization
  2. 2026Robust Successor Features
  3. 2026Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning
  4. 2025Reward-Aware Proto-Representations in Reinforcement Learning
  5. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  6. 2024On the benefits of pixel-based hierarchical policies for task generalization
  7. 2024A Distributional Analogue to the Successor Representation
  8. 2024Learning Successor Features the Simple Way