# Dynamic Evaluation of Neural Sequence Models

arXiv: https://arxiv.org/abs/1709.07432 · HTML: https://mlsanity.com/paper/1709.07432/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2017.

## Built on by (most cited first, 30 of 55)

- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks](https://mlsanity.com/paper/2309.17167/) (2023)
- [Test-Time Training on Nearest Neighbors for Large Language Models](https://mlsanity.com/paper/2305.18466/) (2023)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [End-to-End Test-Time Training for Long Context](https://mlsanity.com/paper/2512.23675/) (2025)
- [MesaNet: Sequence Modeling by Locally Optimal Test-Time Training](https://mlsanity.com/paper/2506.05233/) (2025)
- [Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers](https://mlsanity.com/paper/2101.00234/) (2021)
- [A Modern Self-Referential Weight Matrix That Learns to Modify Itself](https://mlsanity.com/paper/2202.05780/) (2022)
- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://mlsanity.com/paper/2206.06178/) (2022)
- [Latent Thought Models with Variational Bayes Inference-Time Computation](https://mlsanity.com/paper/2502.01567/) (2025)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [Meta-Learning Fast Weight Language Models](https://mlsanity.com/paper/2212.02475/) (2022)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing](https://mlsanity.com/paper/2406.14230/) (2024)
- [A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning](https://mlsanity.com/paper/2205.10787/) (2022)
- [kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Models](https://mlsanity.com/paper/2302.10879/) (2023)
- [The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute](https://mlsanity.com/paper/2309.11197/) (2023)
- [Active Fine-Tuning of Multi-Task Policies](https://mlsanity.com/paper/2410.05026/) (2024)
- [Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging](https://mlsanity.com/paper/2505.14136/) (2025)
- [Test-time Offline Reinforcement Learning on Goal-related Experience](https://mlsanity.com/paper/2507.18809/) (2025)
- [Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models](https://mlsanity.com/paper/2403.01518/) (2024)
- [Construction Repetition Reduces Information Rate in Dialogue](https://mlsanity.com/paper/2210.08321/) (2022)
- [Training Integer-Only Deep Recurrent Neural Networks](https://mlsanity.com/paper/2212.11791/) (2022)
- [Collapse of Self-trained Language Models](https://mlsanity.com/paper/2404.02305/) (2024)

## Abstract (arXiv)

We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.