mlsanityData to 7 Oct 2026

Perceiver IO: A General Architecture for Structured Inputs & Outputs

arXiv 2107.14795 ↗First posted 30 Jul 2021cs.LG cs.CL cs.CVcited by 616

Not on the map. Why?

Cited by 37 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain & task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data from arbitrary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natural language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

Key facts. Cited by 616 papers in the map (data through 7 Oct 2026). Builds on 80 of its 104 references that are in the map. First posted on arXiv 30 Jul 2021.

Builds on

80 of 104 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Mask R-CNNcited by 7,259
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  13. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  14. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  15. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  16. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  17. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  18. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  19. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  20. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  21. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  22. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  23. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  24. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  25. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  26. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  27. 2015Pointer Networkscited by 882
  28. 2020Object-Centric Learning with Slot Attentioncited by 865
  29. 2020Efficient Transformers: A Surveycited by 784
  30. 2021Going deeper with Image Transformerscited by 759
  31. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  32. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  33. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  34. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  35. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  36. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  37. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  38. 2016Cross-stitch Networks for Multi-task Learningcited by 466

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 117; 2023: 130; 2024: 124; 2025: 110; 2026: 119.

616 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2023Vision Transformers Need Registerscited by 844
  4. 2022A Generalist Agentcited by 802
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2023CoTracker: It is Better to Track Togethercited by 516
  11. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  12. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
38 more
  1. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  3. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  4. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  5. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  6. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  7. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  8. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  9. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  10. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  11. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  12. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 273
  13. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  14. 2021Conditional Object-Centric Learning from Videocited by 247
  15. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  16. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  17. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  18. 2022Recurrent Memory Transformercited by 197
  19. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  20. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  21. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  22. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 178
  23. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  24. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  25. 2022A Unified Sequence Interface for Vision Taskscited by 152
  26. 2022Multiview Transformers for Video Recognitioncited by 148
  27. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  28. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  29. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  30. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  31. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  32. 2022Task Residual for Tuning Vision-Language Modelscited by 122
  33. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  34. 20234M: Massively Multimodal Masked Modelingcited by 118
  35. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  36. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimationcited by 116
  37. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  38. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 109

50 most cited of 616 All 616 (JSON)