mlsanityData to 7 Oct 2026

Token Turing Machines

arXiv 2211.09119 ↗First posted 16 Nov 2022cs.LG cs.CV cs.ROcited by 25

Not on the map. Why?

Abstract (arXiv)

We propose Token Turing Machines (TTM), a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history (i.e., frames). This memory is efficiently addressed, read and written using a Transformer as the processing unit/controller at each step. The model's memory module ensures that a new observation will only be processed with the contents of the memory (and not the entire history), meaning that it can efficiently process long sequences with a bounded computational cost at each step. We show that TTM outperforms other alternatives, such as other Transformer models designed for long sequences and recurrent neural networks, on two real-world sequential visual understanding tasks: online temporal activity detection from videos and vision-based robot action policy learning. Code is publicly available at: https://github.com/google-research/scenic/tree/main/scenic/projects/token_turing

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 63 of its 75 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

63 of 75 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2018SlowFast Networks for Video Recognitioncited by 1,862
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  13. 2021Video Swin Transformercited by 1,090
  14. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  15. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  16. 2021Multiscale Vision Transformerscited by 896
  17. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  18. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  19. 2020Efficient Transformers: A Surveycited by 784
  20. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  21. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  22. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  23. 2014Neural Turing Machinescited by 630
  24. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  25. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  26. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  27. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  28. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  29. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  30. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  31. 2021Random Feature Attentioncited by 350
  32. 2018A Short Note about Kinetics-600cited by 331
  33. 2022Memorizing Transformerscited by 219
  34. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  35. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  36. 2022Block-Recurrent Transformerscited by 107
  37. 2015Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videoscited by 93
  38. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 7; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024Streaming Dense Video Captioningcited by 82
  2. 2024Language Repository for Long Video Understandingcited by 55
  3. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  4. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  5. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  6. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  7. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  9. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  10. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  11. 2024Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagationcited by 12
  12. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representationscited by 6
13 more
  1. 2025Long-VMNet: Accelerating Long-Form Video Understanding via Fixed Memorycited by 4
  2. 2024Extending Video Masked Autoencoders to 128 framescited by 3
  3. 2024Token Turing Machines are Efficient Vision Modelscited by 2
  4. 2025MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videoscited by 2
  5. 2025Kinaema: a recurrent sequence model for memory and pose in motioncited by 2
  6. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  7. 2023AAN: Attributes-Aware Network for Temporal Action Detectioncited by 1
  8. 2025VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMscited by 1
  9. 2026FlowNar: Scalable Streaming Narration for Long-Form Videoscited by 1
  10. 2026Compressing History into Memory: Distilling Transformers into Recurrent Transformerscited by 1
  11. 2023Active Vision Reinforcement Learning under Limited Visual Observabilitycited by 0
  12. 2025Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategycited by 0
  13. 2026Efficient Tracking and Understanding Object Transformationscited by 0