# DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales

arXiv: https://arxiv.org/abs/2104.04646 · HTML: https://mlsanity.com/paper/2104.04646/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 41 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies](https://mlsanity.com/paper/2010.00951/) (2020)
- [Recurrent Network Models Of Sequence Generation And Memory](https://mlsanity.com/paper/1603.04687/) (2016)
- [Deep Independently Recurrent Neural Network (IndRNN)](https://mlsanity.com/paper/1910.06251/) (2019)
- [Time-causal and time-recursive spatio-temporal receptive fields](https://mlsanity.com/paper/1504.02648/) (2015)
- [Neural scaling laws for an uncertain world](https://mlsanity.com/paper/1607.04886/) (2016)

## Built on by (most cited first, 5 of 5)

- [Traveling Waves Encode the Recent Past and Enhance Sequence Learning](https://mlsanity.com/paper/2309.08045/) (2023)
- [A deep convolutional neural network that is invariant to time rescaling](https://mlsanity.com/paper/2107.04616/) (2021)
- [Deep reinforcement learning with time-scale invariant memory](https://mlsanity.com/paper/2412.15292/) (2024)
- [Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows](https://mlsanity.com/paper/2510.22109/) (2025)
- [Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks](https://mlsanity.com/paper/2601.02618/) (2026)

## Abstract (arXiv)

Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition -- and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves state-of-the-art performance on these problems.