# Reservoir Transformers

arXiv: https://arxiv.org/abs/2012.15045 · HTML: https://mlsanity.com/paper/2012.15045/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 18 of 18)

- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.20587/) (2023)
- [How does the pre-training objective affect what large language models learn about linguistic properties?](https://mlsanity.com/paper/2203.10415/) (2022)
- [Efficient and Private Federated Learning with Partially Trainable Networks](https://mlsanity.com/paper/2110.03450/) (2021)
- [Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training](https://mlsanity.com/paper/2209.11204/) (2022)
- [What's Hidden in a One-layer Randomly Weighted Transformer?](https://mlsanity.com/paper/2109.03939/) (2021)
- [Partially Randomizing Transformer Weights for Dialogue Response Diversity](https://mlsanity.com/paper/2311.10943/) (2023)
- [Perturbative Gradient Training: A novel training paradigm for bridging the gap between deep neural networks and physical reservoir computing](https://mlsanity.com/paper/2506.04523/) (2025)
- [Frozen in Time: Parameter-Efficient Time Series Transformers via Reservoir-Induced Feature Expansion and Fixed Random Dynamics](https://mlsanity.com/paper/2508.18130/) (2025)
- [Training-Free Universal Approximation by Prompting Random Transformers](https://mlsanity.com/paper/2608.09558/) (2026)
- [Physical Deep Learning with Biologically Plausible Training Method](https://mlsanity.com/paper/2204.13991/) (2022)
- [Changes by Butterflies: Farsighted Forecasting with Group Reservoir Transformer](https://mlsanity.com/paper/2402.09573/) (2024)
- [Federated Koopman-Reservoir Learning for Large-Scale Multivariate Time-Series Anomaly Detection](https://mlsanity.com/paper/2503.11255/) (2025)
- [Learning Music Audio Representations With Limited Data](https://mlsanity.com/paper/2505.06042/) (2025)
- [ResFormer: All-Time Reservoir Memory for Long Sequence Classification](https://mlsanity.com/paper/2509.24074/) (2025)
- [Echo Flow Networks](https://mlsanity.com/paper/2509.24122/) (2025)
- [Reservoir Computing inspired Matrix Multiplication-free Language Model](https://mlsanity.com/paper/2512.23145/) (2025)
- [Quantum Reservoir Computing: Recent Advances and Future Directions](https://mlsanity.com/paper/2607.18552/) (2026)

## Abstract (arXiv)

We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.