# Token Turing Machines

arXiv: https://arxiv.org/abs/2211.09119 · HTML: https://mlsanity.com/paper/2211.09119/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 75 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)

## Built on by (most cited first, 25 of 25)

- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [Language Repository for Long Video Understanding](https://mlsanity.com/paper/2403.14622/) (2024)
- [Memory Consolidation Enables Long-Context Video Understanding](https://mlsanity.com/paper/2402.05861/) (2024)
- [Generalization to New Sequential Decision Making Tasks with In-Context Learning](https://mlsanity.com/paper/2312.03801/) (2023)
- [A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames](https://mlsanity.com/paper/2312.07395/) (2023)
- [xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs](https://mlsanity.com/paper/2410.16267/) (2024)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [A Survey of Token Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2507.20198/) (2025)
- [VideoLucy: Deep Memory Backtracking for Long Video Understanding](https://mlsanity.com/paper/2510.12422/) (2025)
- [Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities](https://mlsanity.com/paper/2311.05698/) (2023)
- [Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagation](https://mlsanity.com/paper/2411.11002/) (2024)
- [xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations](https://mlsanity.com/paper/2408.12590/) (2024)
- [Long-VMNet: Accelerating Long-Form Video Understanding via Fixed Memory](https://mlsanity.com/paper/2503.13707/) (2025)
- [Extending Video Masked Autoencoders to 128 frames](https://mlsanity.com/paper/2411.13683/) (2024)
- [Token Turing Machines are Efficient Vision Models](https://mlsanity.com/paper/2409.07613/) (2024)
- [MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videos](https://mlsanity.com/paper/2501.06138/) (2025)
- [Kinaema: a recurrent sequence model for memory and pose in motion](https://mlsanity.com/paper/2510.20261/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [AAN: Attributes-Aware Network for Temporal Action Detection](https://mlsanity.com/paper/2309.00696/) (2023)
- [VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs](https://mlsanity.com/paper/2512.22226/) (2025)
- [FlowNar: Scalable Streaming Narration for Long-Form Videos](https://mlsanity.com/paper/2606.00620/) (2026)
- [Compressing History into Memory: Distilling Transformers into Recurrent Transformers](https://mlsanity.com/paper/2606.21562/) (2026)
- [Active Vision Reinforcement Learning under Limited Visual Observability](https://mlsanity.com/paper/2306.00975/) (2023)
- [Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy](https://mlsanity.com/paper/2501.18509/) (2025)
- [Efficient Tracking and Understanding Object Transformations](https://mlsanity.com/paper/2607.19743/) (2026)

## Abstract (arXiv)

We propose Token Turing Machines (TTM), a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history (i.e., frames). This memory is efficiently addressed, read and written using a Transformer as the processing unit/controller at each step. The model's memory module ensures that a new observation will only be processed with the contents of the memory (and not the entire history), meaning that it can efficiently process long sequences with a bounded computational cost at each step. We show that TTM outperforms other alternatives, such as other Transformer models designed for long sequences and recurrent neural networks, on two real-world sequential visual understanding tasks: online temporal activity detection from videos and vision-based robot action policy learning. Code is publicly available at: https://github.com/google-research/scenic/tree/main/scenic/projects/token_turing