# Variational Temporal Abstraction

arXiv: https://arxiv.org/abs/1910.00775 · HTML: https://mlsanity.com/paper/1910.00775/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 30 of 38)

- [Flexible Diffusion Modeling of Long Videos](https://mlsanity.com/paper/2205.11495/) (2022)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [Simple Hierarchical Planning with Diffusion](https://mlsanity.com/paper/2401.02644/) (2024)
- [Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models](https://mlsanity.com/paper/2205.13817/) (2022)
- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Subgoal Search For Complex Reasoning Tasks](https://mlsanity.com/paper/2108.11204/) (2021)
- [Deep Explicit Duration Switching Models for Time Series](https://mlsanity.com/paper/2110.13878/) (2021)
- [Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model](https://mlsanity.com/paper/2312.08403/) (2023)
- [Unsupervised Learning of Temporal Abstractions with Slot-based Transformers](https://mlsanity.com/paper/2203.13573/) (2022)
- [Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration](https://mlsanity.com/paper/2410.18076/) (2024)
- [Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search](https://mlsanity.com/paper/2206.00702/) (2022)
- [Language-guided Skill Learning with Temporal Variational Inference](https://mlsanity.com/paper/2402.16354/) (2024)
- [Exploring the limits of Hierarchical World Models in Reinforcement Learning](https://mlsanity.com/paper/2406.00483/) (2024)
- [Variational Predictive Routing with Nested Subjective Timescales](https://mlsanity.com/paper/2110.11236/) (2021)
- [Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Experts](https://mlsanity.com/paper/2402.05970/) (2024)
- [LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following](https://mlsanity.com/paper/2310.12344/) (2023)
- [Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning](https://mlsanity.com/paper/2206.11396/) (2022)
- [Multi-View Dreaming: Multi-View World Model with Contrastive Learning](https://mlsanity.com/paper/2203.11024/) (2022)
- [Hybrid Search for Efficient Planning with Completeness Guarantees](https://mlsanity.com/paper/2310.12819/) (2023)
- [PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer](https://mlsanity.com/paper/2406.06793/) (2024)
- [A Temporal Variational Model for Story Generation](https://mlsanity.com/paper/2109.06807/) (2021)
- [Long-horizon video prediction using a dynamic latent hierarchy](https://mlsanity.com/paper/2212.14376/) (2022)
- [Model-Based Reinforcement Learning with Isolated Imaginations](https://mlsanity.com/paper/2303.14889/) (2023)
- [Identifying Selections for Unsupervised Subtask Discovery](https://mlsanity.com/paper/2410.21616/) (2024)
- [Conditional COT-GAN for Video Prediction with Kernel Smoothing](https://mlsanity.com/paper/2106.05658/) (2021)
- [Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems](https://mlsanity.com/paper/2605.16198/) (2026)

## Abstract (arXiv)

We introduce a variational approach to learning and inference of temporally hierarchical structure and representation for sequential data. We propose the Variational Temporal Abstraction (VTA), a hierarchical recurrent state space model that can infer the latent temporal structure and thus perform the stochastic state transition hierarchically. We also propose to apply this model to implement the jumpy-imagination ability in imagination-augmented agent-learning in order to improve the efficiency of the imagination. In experiments, we demonstrate that our proposed method can model 2D and 3D visual sequence datasets with interpretable temporal structure discovery and that its application to jumpy imagination enables more efficient agent-learning in a 3D navigation task.