Data to 7 Oct 2026

Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

arXiv 1606.01305 ↗First posted 3 Jun 2016cs.NE cs.CL cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

We propose zoneout, a novel method for regularizing RNNs. At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization. But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks. We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 12; 2023: 11; 2024: 2; 2025: 1; 2026: 6.

54 papers.

Sort:

By area:

2023A Comprehensive Overview of Large Language Modelscited by 639
2023End-to-End Speech Recognition: A Surveycited by 127
2022Efficient Transformers with Dynamic Token Poolingcited by 75
2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
2021Long Expressive Memory for Sequence Modelingcited by 41
2021Discrete-Valued Neural Communicationcited by 39
2021On the limit of English conversational speech recognitioncited by 30
2021Librispeech Transducer Model with Internal Language Model Prior Correctioncited by 26
2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
2021Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Modelscited by 25

38 more

2021High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latencycited by 19
2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
2023A Review of the Role of Causality in Developing Trustworthy AI Systemscited by 14
2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
2025Quantifying and Alleviating Co-Adaptation in Sparse-View 3D Gaussian Splattingcited by 12
2021ChiNet: Deep Recurrent Convolutional Learning for Multimodal Spacecraft Pose Estimationcited by 11
2021Distilling Large Language Models into Tiny and Effective Students using pQRNNcited by 10
2021UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Costcited by 10
2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
2021LocalDrop: A Hybrid Regularization for Deep Neural Networkscited by 8
2021Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architecturescited by 8
2021Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesiscited by 8
2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
2024Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learningcited by 7
2021Meta-Forecasting by combining Global Deep Representations with Local Adaptationcited by 5
2022A Survey on Dropout Methods and Experimental Verification in Recommendationcited by 4
2021Noise Injection-based Regularization for Point Cloud Processingcited by 3
2023SeqAug: Sequential Feature Resampling as a modality agnostic augmentation methodcited by 3
2023Modularity in Deep Learning: A Surveycited by 3
2021Preventing posterior collapse in variational autoencoders for text generation via decoder regularizationcited by 2
2021Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Controlcited by 2
2022Zero-Shot Long-Form Voice Cloning with Dynamic Convolution Attentioncited by 2
2022State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictionscited by 2
2023Optimum Output Long Short-Term Memory Cell for High-Frequency Trading Forecastingcited by 2
2021Zero Training Overhead Portfolios for Learning to Solve Combinatorial Problemscited by 1
2022Sparse-Dyn: Sparse Dynamic Graph Multi-representation Learning via Event-based Sparse Temporal Attention Networkcited by 1
2023DropDim: A Regularization Method for Transformer Networkscited by 1
2023Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Trainingcited by 1
2021Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networkscited by 0
2021Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Trainingcited by 0
2022Improving End-to-End Models for Set Prediction in Spoken Language Understandingcited by 0
2022Look Backward and Forward: Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translationcited by 0
2022ZoDIAC: Zoneout Dropout Injection Attention Calculationcited by 0
2022Learning Sequence Representations by Non-local Recurrent Neural Memorycited by 0
2023RigLSTM: Recurrent Independent Grid LSTM for Generalizable Sequence Learningcited by 0
2024Y-Drop: A Conductance based Dropout for fully connected layerscited by 0
2026Rethinking Recurrent Neural Networks for Time Series Forecasting: A Reinforced Recurrent Encoder with Prediction-Oriented Proximal Policy Optimizationcited by 0
2026Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modelingcited by 0

50 most cited of 54 All 54 (JSON)