Can We Really Learn One Representation to Optimize All Rewards?
Not on the map. Why?
Abstract (arXiv)
As unsupervised pretraining becomes increasingly ubiquitous in reinforcement learning, a more thorough theoretical understanding of these methods becomes of equal importance to their empirical success. We focus on the setting of unsupervised learning via interaction, where the forward-backward (FB) representation learning serves as a prototypical and popular example. In this paper, we shed light on FB by formally contextualizing the method within a broader class of recent methods that use regression to obtain a low-rank approximation of a successor measure ratio. Our analysis clarifies when FB representations can exist and how the low-rank approximation converges in practice. Building upon the theory, we propose a variant of FB that is both more amenable to theoretical understanding and simpler to optimize in practice. Experiments in didactic settings, as well as in 10 state-based and image-based continuous control domains, demonstrate that our method converges to desired representations with 105 × smaller errors than FB, achieving +24% improved zero-shot performance on average. We also demonstrate that zero-shot policies inferred by our algorithm provide an efficient initialization if the user prefers further fine-tuning on downstream tasks. Our project website is available at https://chongyi-zheng.github.io/onestep-fb.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 73 of its 116 references that are in the map. First posted on arXiv 11 Feb 2026.
Builds on
73 of 116 references.
References · 73
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
38 more
- 2018Exploration by Random Network Distillationcited by 966
- 2018DeepMind Control Suitecited by 935
- 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2018MINE: Mutual Information Neural Estimationcited by 680
- 2019On Variational Bounds of Mutual Informationcited by 550
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
- 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
- 2016Successor Features for Transfer in Reinforcement Learningcited by 331
- 2016Revisiting Classifier Two-Sample Testscited by 258
- 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
- 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
- 2018Semi-parametric Topological Memory for Navigationcited by 203
- 2025Flow Q-Learningcited by 162
- 2019Goal-conditioned Imitation Learningcited by 159
- 2021Offline RL Without Off-Policy Evaluationcited by 150
- 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
- 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
- 2021Learning One Representation to Optimize All Rewardscited by 110
- 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
- 2018Universal Successor Features Approximatorscited by 102
- 2019Reward-Conditioned Policiescited by 101
- 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
- 2024Foundation Policies with Hilbert Representationscited by 89
- 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
- 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
- 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
- 2016Deep Successor Reinforcement Learningcited by 72
- 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
- 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
- 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
- 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
- 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
- 2025Horizon Reduction Makes RL Scalablecited by 55
- 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
50 most cited of 73 All 73 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.