# Z-Forcing: Training Stochastic Recurrent Networks

arXiv: https://arxiv.org/abs/1711.05411 · HTML: https://mlsanity.com/paper/1711.05411/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2017.

## Built on by (most cited first, 30 of 51)

- [Modeling Irregular Time Series with Continuous Recurrent Units](https://mlsanity.com/paper/2111.11344/) (2021)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [SocialVAE: Human Trajectory Prediction using Timewise Latents](https://mlsanity.com/paper/2203.08207/) (2022)
- [Guiding Language Model Reasoning with Planning Tokens](https://mlsanity.com/paper/2310.05707/) (2023)
- [Causal Recurrent Variational Autoencoder for Medical Time Series Generation](https://mlsanity.com/paper/2301.06574/) (2023)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [Learning Belief Representations for Imitation Learning in POMDPs](https://mlsanity.com/paper/1906.09510/) (2019)
- [GLENet: Boosting 3D Object Detectors with Generative Label Uncertainty Estimation](https://mlsanity.com/paper/2207.02466/) (2022)
- [Evaluation of creating scoring opportunities for teammates in soccer via trajectory prediction](https://mlsanity.com/paper/2206.01899/) (2022)
- [Data-to-text Generation with Variational Sequential Planning](https://mlsanity.com/paper/2202.13756/) (2022)
- [CLVSA: A Convolutional LSTM Based Variational Sequence-to-Sequence Model with Attention for Predicting Trends of Financial Markets](https://mlsanity.com/paper/2104.04041/) (2021)
- [Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries](https://mlsanity.com/paper/2510.14751/) (2025)
- [FutureDepth: Learning to Predict the Future Improves Video Depth Estimation](https://mlsanity.com/paper/2403.12953/) (2024)
- [Estimating counterfactual treatment outcomes over time in complex multiagent scenarios](https://mlsanity.com/paper/2206.01900/) (2022)
- [GePSAn: Generative Procedure Step Anticipation in Cooking Videos](https://mlsanity.com/paper/2310.08312/) (2023)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)
- [LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies](https://mlsanity.com/paper/2108.13459/) (2021)
- [Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection](https://mlsanity.com/paper/2106.12382/) (2021)
- [DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards](https://mlsanity.com/paper/2304.10770/) (2023)
- [Learning Physics for Unveiling Hidden Earthquake Ground Motions via Conditional Generative Modeling](https://mlsanity.com/paper/2407.15089/) (2024)
- [VDSM: Unsupervised Video Disentanglement with State-Space Modeling and Deep Mixtures of Experts](https://mlsanity.com/paper/2103.07292/) (2021)
- [Policy Gradients Incorporating the Future](https://mlsanity.com/paper/2108.02096/) (2021)
- [Competency Assessment for Autonomous Agents using Deep Generative Models](https://mlsanity.com/paper/2203.12670/) (2022)
- [Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation](https://mlsanity.com/paper/2505.17099/) (2025)
- [AS-IntroVAE: Adversarial Similarity Distance Makes Robust IntroVAE](https://mlsanity.com/paper/2206.13903/) (2022)
- [Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory](https://mlsanity.com/paper/2103.03905/) (2021)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Deep Generative Modeling for Identification of Noisy, Non-Stationary Dynamical Systems](https://mlsanity.com/paper/2410.02079/) (2024)

## Abstract (arXiv)

Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We unify successful ideas from recently proposed architectures into a stochastic recurrent model: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortized variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. We found this strategy to perform better than alternative approaches such as KL annealing. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and Blizzard and competitive performance on sequential MNIST. Finally, we apply our model to language modeling on the IMDB dataset where the auxiliary cost helps in learning interpretable latent variables. Source Code: urlhttps://github.com/anirudh9119/zforcingnips17