# World Value Functions: Knowledge Representation for Learning and Planning

arXiv: https://arxiv.org/abs/2206.11940 · HTML: https://mlsanity.com/paper/2206.11940/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 11 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 3 of 3)

- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [A Boolean Task Algebra for Reinforcement Learning](https://mlsanity.com/paper/2001.01394/) (2020)
- [Many-Goals Reinforcement Learning](https://mlsanity.com/paper/1806.09605/) (2018)

## Built on by (most cited first, 1 of 1)

- [Scaling Goal-based Exploration via Pruning Proto-goals](https://mlsanity.com/paper/2302.04693/) (2023)

## Abstract (arXiv)

We propose world value functions (WVFs), a type of goal-oriented general value function that represents how to solve not just a given task, but any other goal-reaching task in an agent's environment. This is achieved by equipping an agent with an internal goal space defined as all the world states where it experiences a terminal transition. The agent can then modify the standard task rewards to define its own reward function, which provably drives it to learn how to achieve all reachable internal goals, and the value of doing so in the current task. We demonstrate two key benefits of WVFs in the context of learning and planning. In particular, given a learned WVF, an agent can compute the optimal policy in a new task by simply estimating the task's reward function. Furthermore, we show that WVFs also implicitly encode the transition dynamics of the environment, and so can be used to perform planning. Experimental results show that WVFs can be learned faster than regular value functions, while their ability to infer the environment's dynamics can be used to integrate learning and planning methods to further improve sample efficiency.