# Predictive-State Decoders: Encoding the Future into Recurrent Networks

arXiv: https://arxiv.org/abs/1709.08520 · HTML: https://mlsanity.com/paper/1709.08520/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2017.

## Built on by (most cited first, 11 of 11)

- [PAC Reinforcement Learning for Predictive State Representations](https://mlsanity.com/paper/2207.05738/) (2022)
- [Learning Belief Representations for Imitation Learning in POMDPs](https://mlsanity.com/paper/1906.09510/) (2019)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Neural Recursive Belief States in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.02274/) (2021)
- [Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning](https://mlsanity.com/paper/2206.11396/) (2022)
- [Pretraining Recurrent Networks without Recurrence](https://mlsanity.com/paper/2606.06479/) (2026)
- [Belief States for Cooperative Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2504.08417/) (2025)
- [Descriptive History Representations: Learning Representations by Answering Questions](https://mlsanity.com/paper/2506.02125/) (2025)
- [Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2605.11880/) (2026)
- [What Can a Recurrent State Safely Forget?](https://mlsanity.com/paper/2609.23366/) (2026)
- [Generative sequence modeling for infinite memory processes via predictive states](https://mlsanity.com/paper/2609.38524/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network's internal state representation to target predicting future observations. Predictive-State Decoders are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.