mlsanityData to 14 Oct 2026

Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

arXiv 1708.06834 ↗First posted 22 Aug 2017cs.AI cs.CVcited by 49

Not on the map. Why?

Abstract (arXiv)

Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/ .

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 13; 2023: 4; 2024: 5; 2025: 3; 2026: 5.

49 papers.

Sort:

By area:

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  2. 2021Dynamic Neural Networks: A Surveycited by 408
  3. 2021PonderNet: Learning to Pondercited by 168
  4. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  5. 2021Closed-form Continuous-time Neural Modelscited by 65
  6. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  7. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  8. 2021A Survey on Green Deep Learningcited by 52
  9. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  10. 2021Modeling Multi-Label Action Dependencies for Temporal Action Localizationcited by 32
  11. 2021Skip-Convolutions for Efficient Video Processingcited by 32
  12. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queriescited by 29
37 more
  1. 2021Variable-rate discrete representation learningcited by 24
  2. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  3. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 24
  4. 2021Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domainscited by 18
  5. 2021Block-Skim: Efficient Question Answering for Transformercited by 18
  6. 2022Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selectioncited by 18
  7. 2022Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routingcited by 18
  8. 2021Learning Asynchronous and Sparse Human-Object Interaction in Videoscited by 16
  9. 2022TSFool: Crafting Highly-Imperceptible Adversarial Time Series through Multi-Objective Attackcited by 9
  10. 2022Learning Robust Dynamics through Variational Sparse Gatingcited by 8
  11. 2021Doping: A technique for efficient compression of LSTM models using sparse structured additive matricescited by 5
  12. 2021Gradient Forward-Propagation for Large-Scale Temporal Video Modellingcited by 5
  13. 2021Event Neural Networkscited by 5
  14. 2022Sequence Learning Using Equilibrium Propagationcited by 5
  15. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  16. 2024Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancementcited by 5
  17. 2023iLoRE: Dynamic Graph Representation with Instant Long-term Modeling and Re-occurrence Preservationcited by 4
  18. 2022Triangular Dropout: Variable Network Width without Retrainingcited by 3
  19. 2021Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processingcited by 2
  20. 2021Part & Whole Extraction: Towards A Deep Understanding of Quantitative Facts for Percentages in Textcited by 2
  21. 2022State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictionscited by 2
  22. 2025Attention when you needcited by 2
  23. 2022VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognitioncited by 1
  24. 2023Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Trainingcited by 1
  25. 2023No-Skim: Towards Efficiency Robustness Evaluation on Skimming-based Language Modelscited by 1
  26. 2025Rethinking Time Encoding via Learnable Transformation Functionscited by 1
  27. 2026Beyond the Clock: Measuring the Value of Adaptive Revisioncited by 1
  28. 2021Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classificationcited by 0
  29. 2022An end-to-end multi-scale network for action prediction in videoscited by 0
  30. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  31. 2024Recent Trends in Modelling the Continuous Time Series using Deep Learning: A Surveycited by 0
  32. 2024SOI: Scaling Down Computational Complexity by Estimating Partial States of the Modelcited by 0
  33. 2024SkipSNN: Efficiently Classifying Spike Trains with Event-attentioncited by 0
  34. 2025Temporal Chunking Enhances Recognition of Implicit Sequential Patternscited by 0
  35. 2026Rethinking Recurrent Neural Networks for Time Series Forecasting: A Reinforced Recurrent Encoder with Prediction-Oriented Proximal Policy Optimizationcited by 0
  36. 2026Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modelingcited by 0
  37. 2026Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligencecited by 0