mlsanityData to 7 Oct 2026

Perceiver: General Perception with Iterative Attention

arXiv 2103.03206 ↗First posted 4 Mar 2021cs.CV cs.AI cs.LGcited by 1,089

Not on the map. Why?

Cited by 84 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver - a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture is competitive with or outperforms strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video, and video+audio. The Perceiver obtains performance comparable to ResNet-50 and ViT on ImageNet without 2D convolutions by directly attending to 50,000 pixels. It is also competitive in all modalities in AudioSet.

Key facts. Cited by 1,089 papers in the map (data through 7 Oct 2026). Builds on 78 of its 117 references that are in the map. First posted on arXiv 4 Mar 2021.

Builds on

78 of 117 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2016Layer Normalizationcited by 5,579
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  13. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  14. 2016Group Equivariant Convolutional Networkscited by 1,154
  15. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  16. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  17. 2020Object-Centric Learning with Slot Attentioncited by 865
  18. 2020Efficient Transformers: A Surveycited by 784
  19. 2018Image Transformercited by 770
  20. 2020PCT: Point cloud transformercited by 730
  21. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  22. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  23. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  24. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  25. 2014Neural Turing Machinescited by 630
  26. 2018Universal Transformerscited by 613
  27. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  28. 2017Convolutional Sequence to Sequence Learningcited by 550
  29. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  30. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  31. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  32. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  33. 2019Attention Augmented Convolutional Networkscited by 363
  34. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  35. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  36. 2014Memory Networkscited by 354
  37. 2019Axial Attention in Multidimensional Transformerscited by 350
  38. 2021Random Feature Attentioncited by 350

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 157; 2023: 202; 2024: 213; 2025: 231; 2026: 236.

1,089 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2023Vision Transformers Need Registerscited by 844
  5. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  11. 2023ModelScope Text-to-Video Technical Reportcited by 651
  12. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
38 more
  1. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  2. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  3. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  4. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  5. 2025Continuous 3D Perception Model with Persistent Statecited by 527
  6. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  7. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  8. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 449
  9. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2022Multimodal Learning with Transformers: A Surveycited by 399
  12. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  13. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  14. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  15. 2024What matters when building vision-language models?cited by 321
  16. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  17. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  18. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  19. 2022Vision Transformer with Deformable Attentioncited by 293
  20. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  21. 2023Towards Generalist Biomedical AIcited by 286
  22. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  23. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  24. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  25. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  26. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  27. 2021Conditional Object-Centric Learning from Videocited by 247
  28. 2023From Sparse to Soft Mixtures of Expertscited by 245
  29. 2024Generative Representational Instruction Tuningcited by 227
  30. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  31. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  32. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  33. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  34. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  35. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  36. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  37. 2022Continuous diffusion for categorical datacited by 202
  38. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201

50 most cited of 1,089 All 1,089 (JSON)