# Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks

arXiv: https://arxiv.org/abs/2601.02618 · HTML: https://mlsanity.com/paper/2601.02618/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 86 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [Were RNNs All We Needed?](https://mlsanity.com/paper/2410.01201/) (2024)
- [Recurrent Network Models Of Sequence Generation And Memory](https://mlsanity.com/paper/1603.04687/) (2016)
- [DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales](https://mlsanity.com/paper/2104.04646/) (2021)
- [A deep convolutional neural network that is invariant to time rescaling](https://mlsanity.com/paper/2107.04616/) (2021)
- [Distributions of covariances as a window into the operational regime of neuronal networks](https://mlsanity.com/paper/1605.04153/) (2016)
- [Cognitive computation using neural representations of time and space in the Laplace domain](https://mlsanity.com/paper/2003.11668/) (2020)
- [Continuous Attractor Networks for Laplace Neural Manifolds](https://mlsanity.com/paper/2406.04545/) (2024)
- ["What" x "When" working memory representations using Laplace Neural Manifolds](https://mlsanity.com/paper/2409.20484/) (2024)
- [Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows](https://mlsanity.com/paper/2510.22109/) (2025)

## Abstract (arXiv)

Human cognition integrates information across nested timescales. While the cortex exhibits hierarchical Temporal Receptive Windows (TRWs), local circuits often display heterogeneous time constants. To reconcile this, we trained biologically constrained deep networks, based on scale-invariant hippocampal time cells, on a language classification task mimicking the hierarchical structure of language (e.g., 'letters' forming 'words'). First, using a feedforward model (SITHCon), we found that a hierarchy of TRWs emerged naturally across layers, despite the network having an identical spectrum of time constants within layers. We then distilled these inductive priors into a biologically plausible recurrent architecture, SITH-RNN. Training a sequence of architectures ranging from generic RNNs to this restricted subset showed that the scale-invariant SITH-RNN learned faster with orders-of-magnitude fewer parameters, and generalized zero-shot to out-of-distribution timescales. These results suggest the brain employs scale-invariant, sequential priors - coding "what" happened "when" - making recurrent networks with such priors particularly well-suited to describe human cognition.