mlsanityData to 7 Oct 2026

General-purpose, long-context autoregressive modeling with Perceiver AR

arXiv 2202.07765 ↗First posted 15 Feb 2022cs.LG cs.AI cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64 x 64 ImageNet images and PG-19 books.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 59 of its 77 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

59 of 77 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  13. 2020On Layer Normalization in the Transformer Architecturecited by 906
  14. 2016Pixel Recurrent Neural Networkscited by 814
  15. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  16. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  17. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  18. 2014Neural Turing Machinescited by 630
  19. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  20. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  21. 2020Jukebox: A Generative Model for Musiccited by 535
  22. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  23. 2014Memory Networkscited by 354
  24. 2021Random Feature Attentioncited by 350
  25. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  26. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  27. 2021Generating Images with Sparse Representationscited by 262
  28. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  29. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  30. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  31. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  32. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  33. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  34. 2018Music Transformercited by 207
  35. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  36. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  37. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  38. 2021Self-attention Does Not Need O(n2) Memorycited by 152

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 16; 2024: 9; 2025: 10; 2026: 9.

60 papers.

Sort:

By area:

  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  4. 2023MusicLM: Generating Music From Textcited by 442
  5. 2022Long Range Language Modeling via Gated State Spacescited by 265
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  7. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
  8. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  9. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  10. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  11. 2022Block-Recurrent Transformerscited by 107
  12. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
38 more
  1. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  2. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  3. 2023SingSong: Generating musical accompaniments from singingcited by 54
  4. 2023Effectively Modeling Time Series with Simple Discrete State Spacescited by 54
  5. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  6. 2025Fractal Generative Modelscited by 44
  7. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  8. 2022Temporally Consistent Transformers for Video Generationcited by 43
  9. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  10. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  11. 2023Anticipatory Music Transformercited by 35
  12. 2024Foundation Models for Music: A Surveycited by 34
  13. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  14. 2023Controllable Music Production with Diffusion Models and Guidance Gradientscited by 29
  15. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  16. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  17. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  18. 2023FIT: Far-reaching Interleaved Transformerscited by 16
  19. 2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
  20. 2022GeneFormer: Learned Gene Compression using Transformer-based Context Modelingcited by 10
  21. 2023Brain2Music: Reconstructing Music from Human Brain Activitycited by 10
  22. 2026Image Generation with a Sphere Encodercited by 8
  23. 2022Neural Attentive Circuitscited by 6
  24. 2025Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformerscited by 6
  25. 2022Is Encoder-Decoder Redundant for Neural Machine Translation?cited by 5
  26. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  27. 2025MEG-GPT: A transformer-based foundation model for magnetoencephalography datacited by 5
  28. 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
  29. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
  30. 2026Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulationcited by 4
  31. 2022A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectivescited by 3
  32. 2023Hierarchical Attention Encoder Decodercited by 3
  33. 2026Proxy Compression for Language Modelingcited by 3
  34. 2022COPER: Continuous Patient State Perceivercited by 2
  35. 2022Efficient Speech Translation with Dynamic Latent Perceiverscited by 2
  36. 2023ACA-Net: Towards Lightweight Speaker Verification using Asymmetric Cross Attentioncited by 1
  37. 2024Adaptive Large Language Models By Layerwise Attention Shortcutscited by 1
  38. 2024Efficient Context Propagating Perceiver Architectures for Auto-Regressive Language Modelingcited by 1

50 most cited of 60 All 60 (JSON)