mlsanityData to 7 Oct 2026

TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

arXiv 2106.11297 ↗First posted 21 Jun 2021cs.CV cs.LGcited by 127

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks. Instead of relying on hand-designed splitting strategies to obtain visual tokens and processing a large number of densely sampled patches for attention, our approach learns to mine important tokens in visual data. This results in efficiently and effectively finding a few important visual tokens and enables modeling of pairwise attention between such tokens, over a longer temporal horizon for videos, or the spatial content in images. Our experiments demonstrate strong performance on several challenging benchmarks for both image and video recognition tasks. Importantly, due to our tokens being adaptive, we accomplish competitive results at significantly reduced compute amount. We obtain comparable results to the state-of-the-arts on ImageNet while being computationally more efficient. We also confirm the effectiveness of the approach on multiple video datasets, including Kinetics-400, Kinetics-600, Charades, and AViD. The code is available at: https://github.com/google-research/scenic/tree/main/scenic/projects/token_learner

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 56 of its 64 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

56 of 64 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
38 more
  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  2. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  6. 2021Scaling Vision Transformerscited by 993
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2021Multiscale Vision Transformerscited by 896
  9. 2021Going deeper with Image Transformerscited by 759
  10. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  11. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  12. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  13. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  14. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  15. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  16. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  17. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  18. 2020Are we done with ImageNet?cited by 329
  19. 2017Temporal Relational Reasoning in Videoscited by 324
  20. 2018Video Action Transformer Networkcited by 316
  21. 2020Exploring Self-attention for Image Recognitioncited by 313
  22. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  23. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  24. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  25. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  26. 2018Videos as Space-Time Region Graphscited by 221
  27. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  28. 2019Scaling Autoregressive Video Modelscited by 189
  29. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  30. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  31. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
  32. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  33. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  34. 2018Timeception for Complex Action Recognitioncited by 89
  35. 2020Evolving Losses for Unsupervised Video Representation Learningcited by 74
  36. 2018D3D: Distilled 3D Networks for Video Action Recognitioncited by 60
  37. 2019AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architecturescited by 46
  38. 2014Learning Spatiotemporal Features with 3D Convolutional Networkscited by 43

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 29; 2023: 31; 2024: 16; 2025: 19; 2026: 20.

127 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  5. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  6. 2023From Sparse to Soft Mixtures of Expertscited by 245
  7. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  8. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  9. 2022DaViT: Dual Attention Vision Transformerscited by 174
  10. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  11. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  12. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
38 more
  1. 2022Multiview Transformers for Video Recognitioncited by 148
  2. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  3. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  4. 2024Streaming Dense Video Captioningcited by 82
  5. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  6. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  7. 2021Self-supervised Video Transformercited by 64
  8. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  9. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  10. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  11. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  12. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  13. 2022MatteFormer: Transformer-Based Image Matting via Prior-Tokenscited by 48
  14. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48
  15. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  16. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  17. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  18. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  19. 2021Co-training Transformer with Videos and Images Improves Action Recognitioncited by 41
  20. 2024MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Maskscited by 39
  21. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  22. 2022Multi-Granularity Prediction for Scene Text Recognitioncited by 36
  23. 2022NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentationcited by 33
  24. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  25. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  26. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  27. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  28. 2022Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answeringcited by 24
  29. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  30. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  31. 2023How can objects help action recognition?cited by 21
  32. 2021Video-Text Pre-training with Learned Regionscited by 20
  33. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformerscited by 19
  34. 2024Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral Image Classificationcited by 18
  35. 2023FIT: Far-reaching Interleaved Transformerscited by 16
  36. 2023Rejuvenating image-GPT as Strong Visual Representation Learnerscited by 16
  37. 2024LookupViT: Compressing visual information to a limited number of tokenscited by 16
  38. 2022The Wisdom of Crowds: Temporal Progressive Attention for Early Action Predictioncited by 13

50 most cited of 127 All 127 (JSON)