mlsanityData to 14 Oct 2026

Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities

arXiv 2602.06769 ↗First posted 6 Feb 2026cs.LGcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recent advancements in zero-shot reinforcement learning (RL) have facilitated the extraction of diverse behaviors from unlabeled, offline data sources. In particular, forward-backward algorithms (FB) can retrieve a family of policies that can approximately solve any standard RL problem (with additive rewards, linear in the occupancy measure), given sufficient capacity. While retaining zero-shot properties, we tackle the greater problem class of RL with general utilities, in which the objective is an arbitrary differentiable function of the occupancy measure. This setting is strictly more expressive, capturing tasks such as distribution matching or pure exploration, which may not be reduced to additive rewards. We show that this additional complexity can be captured by a novel, maximum entropy (soft) variant of the forward-backward algorithm, which recovers a family of stochastic policies from offline data. When coupled with zero-order search over compact policy embeddings, this algorithm can sidestep iterative optimization schemes, and optimizes general utilities directly at test-time. Across both didactic and high-dimensional experiments, we demonstrate that our method retains favorable properties of FB algorithms, while also extending their range to more general RL problems.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 59 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

43 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Optimistic Task Inference for Behavior Foundation Models23 Oct
  2. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning1 Oct
  3. 2025Epistemically-guided forward-backward exploration7 Jul
  4. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models15 Apr
  5. 2025Fast Adaptation with Behavioral Foundation Models10 Apr
  6. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weighting5 Dec
  7. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings27 Feb

Other references · 36

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2018DeepMind Control Suitecited by 956
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  11. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  12. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
24 more
  1. 2021Learning One Representation to Optimize All Rewardscited by 115
  2. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  3. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  4. 2019Recent Advances in Imitation Learning from Observationcited by 93
  5. 2021Reward is enough for convex MDPscited by 84
  6. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  7. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  8. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  9. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  10. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  11. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  12. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 42
  13. 2021Regularized Policies are Reward Robustcited by 32
  14. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  15. 2025Temporal Difference Flowscited by 24
  16. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  17. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  18. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  19. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15
  20. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  21. 2022Policy Gradient for Reinforcement Learning with General Utilitiescited by 6
  22. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learningcited by 6
  23. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  24. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learningcited by 1
  2. 2026It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learningcited by 0
  3. 2026Task Inference Beyond Least Squares in Behavioral Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Robust Zero-Shot Reinforcement Learning
  2. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  3. 2025A Unified Framework for Zero-Shot Reinforcement Learning
  4. 2026Can We Really Learn One Representation to Optimize All Rewards?
  5. 2023Zero-Shot Reinforcement Learning from Low Quality Data
  6. 2025Tackling the Zero-Shot Reinforcement Learning Loss Directly
  7. 2025Zero-Shot Reinforcement Learning Under Partial Observability
  8. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling