# Zero-Shot Reinforcement Learning via Function Encoders

arXiv: https://arxiv.org/abs/2401.17173 · HTML: https://mlsanity.com/paper/2401.17173/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Emergent Complexity via Multi-Agent Competition](https://mlsanity.com/paper/1710.03748/) (2017)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Contextualize Me -- The Case for Context in Reinforcement Learning](https://mlsanity.com/paper/2202.04500/) (2022)
- [Transformers are Meta-Reinforcement Learners](https://mlsanity.com/paper/2206.06614/) (2022)
- [Hidden Parameter Recurrent State Space Models For Changing Dynamics Scenarios](https://mlsanity.com/paper/2206.14697/) (2022)

## Built on by (most cited first, 18 of 18)

- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Neural Chaos: A Spectral Stochastic Neural Operator](https://mlsanity.com/paper/2502.11835/) (2025)
- [Basis-to-Basis Operator Learning Using Function Encoders](https://mlsanity.com/paper/2410.00171/) (2024)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces](https://mlsanity.com/paper/2501.18373/) (2025)
- [Towards Robust Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.15382/) (2025)
- [MoS-VLA: A Vision-Language-Action Model with One-Shot Skill Adaptation](https://mlsanity.com/paper/2510.16617/) (2025)
- [Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shifts](https://mlsanity.com/paper/2506.11033/) (2025)
- [Function Spaces Without Kernels: Learning Compact Hilbert Space Representations](https://mlsanity.com/paper/2509.20605/) (2025)
- [Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed](https://mlsanity.com/paper/2601.21094/) (2026)
- [Reward-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2603.05066/) (2026)
- [Improving Zero-Shot Offline RL via Behavioral Task Sampling](https://mlsanity.com/paper/2604.25496/) (2026)
- [Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning](https://mlsanity.com/paper/2605.25267/) (2026)
- [A Resolution Independent Neural Operator](https://mlsanity.com/paper/2407.13010/) (2024)
- [Zero-Shot Function Encoder-Based Differentiable Predictive Control](https://mlsanity.com/paper/2511.05757/) (2025)
- [Beyond Multimodal Alignment: Certifying Physical Language through Response Substitution and Ordered Execution](https://mlsanity.com/paper/2608.19492/) (2026)
- [Robust Successor Features](https://mlsanity.com/paper/2609.31016/) (2026)
- [Universal Sample-Based Embeddings of Higher-Order Objects](https://mlsanity.com/paper/2610.19899/) (2026)

## Abstract (arXiv)

Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge. The difficulty lies in finding a good representation for the current task so that the agent understands how it relates to previously seen tasks. To achieve zero-shot transfer, we introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions. By using a function encoder to represent the reward function or the transition function, the agent has information on how the current task relates to previously seen tasks via a coherent vector representation. Thus, the agent is able to achieve transfer between related tasks at run time with no additional training. We demonstrate state-of-the-art data efficiency, asymptotic performance, and training stability in three RL fields by augmenting basic RL algorithms with a function encoder task representation.