# Temporal Difference Variational Auto-Encoder

arXiv: https://arxiv.org/abs/1806.03107 · HTML: https://mlsanity.com/paper/1806.03107/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2018.

## Built on by (most cited first, 30 of 49)

- [DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2308.07498/) (2023)
- [StretchBEV: Stretching Future Instance Prediction Spatially and Temporally](https://mlsanity.com/paper/2203.13641/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [SLAMP: Stochastic Latent Appearance and Motion Prediction](https://mlsanity.com/paper/2108.02760/) (2021)
- [Action-Sufficient State Representation Learning for Control with Structural Constraints](https://mlsanity.com/paper/2110.05721/) (2021)
- [Learning Belief Representations for Imitation Learning in POMDPs](https://mlsanity.com/paper/1906.09510/) (2019)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Online Variational Filtering and Parameter Learning](https://mlsanity.com/paper/2110.13549/) (2021)
- [Abstraction for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.05839/) (2022)
- [Neural Recursive Belief States in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.02274/) (2021)
- [Flow-based Recurrent Belief State Learning for POMDPs](https://mlsanity.com/paper/2205.11051/) (2022)
- [Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models](https://mlsanity.com/paper/2101.07046/) (2021)
- [Unsupervised Learning of Temporal Abstractions with Slot-based Transformers](https://mlsanity.com/paper/2203.13573/) (2022)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data](https://mlsanity.com/paper/2406.17768/) (2024)
- [Training a Resilient Q-Network against Observational Interference](https://mlsanity.com/paper/2102.09677/) (2021)
- [Variational Predictive Routing with Nested Subjective Timescales](https://mlsanity.com/paper/2110.11236/) (2021)
- [The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Models](https://mlsanity.com/paper/2303.03284/) (2023)
- [Strata-NeRF : Neural Radiance Fields for Stratified Scenes](https://mlsanity.com/paper/2308.10337/) (2023)
- [VDSM: Unsupervised Video Disentanglement with State-Space Modeling and Deep Mixtures of Experts](https://mlsanity.com/paper/2103.07292/) (2021)
- [Competency Assessment for Autonomous Agents using Deep Generative Models](https://mlsanity.com/paper/2203.12670/) (2022)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [A Temporal Variational Model for Story Generation](https://mlsanity.com/paper/2109.06807/) (2021)
- [Learning Abstract World Model for Value-preserving Planning with Options](https://mlsanity.com/paper/2406.15850/) (2024)
- [Representation Learning For Efficient Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.02890/) (2024)
- [Versatile Physics-based Character Control with Hybrid Latent Representation](https://mlsanity.com/paper/2503.12814/) (2025)
- [Piecewise-constant Neural ODEs](https://mlsanity.com/paper/2106.06621/) (2021)
- [Q-learning with online random forests](https://mlsanity.com/paper/2204.03771/) (2022)
- [DIDER: Discovering Interpretable Dynamically Evolving Relations](https://mlsanity.com/paper/2208.10592/) (2022)

## Abstract (arXiv)

To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.