# Goals and the Structure of Experience

arXiv: https://arxiv.org/abs/2508.15013 · HTML: https://mlsanity.com/paper/2508.15013/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 23 of its 116 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 23 of 23)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning](https://mlsanity.com/paper/2008.00614/) (2020)
- [Technological Approach to Mind Everywhere (TAME): an experimentally-grounded framework for understanding diverse bodies and minds](https://mlsanity.com/paper/2201.10346/) (2021)
- [Decoupling Dynamics and Reward for Transfer Learning](https://mlsanity.com/paper/1804.10689/) (2018)
- [Cognitive science as a source of forward and inverse models of human decisions for robotics and control](https://mlsanity.com/paper/2109.00127/) (2021)
- [Learning Task-Driven Control Policies via Information Bottlenecks](https://mlsanity.com/paper/2002.01428/) (2020)
- [Intrinsic Motivation in Dynamical Control Systems](https://mlsanity.com/paper/2301.00005/) (2022)
- [Predictive representations: building blocks of intelligence](https://mlsanity.com/paper/2402.06590/) (2024)
- [Bounding the Optimal Value Function in Compositional Reinforcement Learning](https://mlsanity.com/paper/2303.02557/) (2023)
- [Conditions on Preference Relations that Guarantee the Existence of Optimal Policies](https://mlsanity.com/paper/2311.01990/) (2023)
- [Task-Driven Estimation and Control via Information Bottlenecks](https://mlsanity.com/paper/1809.07874/) (2018)
- [Multi-Objective Policy Gradients with Topological Constraints](https://mlsanity.com/paper/2209.07096/) (2022)
- [States as goal-directed concepts: an epistemic approach to state-representation learning](https://mlsanity.com/paper/2312.02367/) (2023)
- [Goal-Driven Cognition in the Brain: A Computational Framework](https://mlsanity.com/paper/1404.7591/) (2014)

## Abstract (arXiv)

Purposeful behavior is a hallmark of natural and artificial intelligence. Its acquisition is often believed to rely on world models, comprising both descriptive (what is) and prescriptive (what is desirable) aspects that identify and evaluate state of affairs in the world, respectively. Canonical computational accounts of purposeful behavior, such as reinforcement learning, posit distinct components of a world model comprising a state representation (descriptive aspect) and a reward function (prescriptive aspect). However, an alternative possibility, which has not yet been computationally formulated, is that these two aspects instead co-emerge interdependently from an agent's goal. Here, we describe a computational framework of goal-directed state representation in cognitive agents, in which the descriptive and prescriptive aspects of a world model co-emerge from agent-environment interaction sequences, or experiences. Drawing on Buddhist epistemology, we introduce a construct of goal-directed, or telic, states, defined as classes of goal-equivalent experience distributions. Telic states provide a parsimonious account of goal-directed learning in terms of the statistical divergence between behavioral policies and desirable experience features. We review empirical and theoretical literature supporting this novel perspective and discuss its potential to provide a unified account of behavioral, phenomenological and neural dimensions of purposeful behaviors across diverse substrates.