mlsanityData to 7 Oct 2026

Pretrained Transformers as Universal Computation Engines

arXiv 2103.05247 ↗First posted 9 Mar 2021cs.LG cs.AIcited by 162

Not on the map. Why?

Abstract (arXiv)

We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks. We consider such a model, which we call a Frozen Pretrained Transformer (FPT), and study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. In contrast to prior works which investigate finetuning on the same modality as the pretraining dataset, we show that pretraining on natural language can improve performance and compute efficiency on non-language downstream tasks. Additionally, we perform an analysis of the architecture, comparing the performance of a random initialized transformer to a random LSTM. Combining the two insights, we find language-pretrained transformers can obtain strong performance on a variety of non-language tasks.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Builds on 45 of its 79 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

45 of 79 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2016Layer Normalizationcited by 5,579
33 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  8. 2019Deep Equilibrium Modelscited by 603
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2017Learning multiple visual domains with residual adapterscited by 553
  11. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  12. 2020Hopfield Networks is All You Needcited by 461
  13. 2016Adversarial Feature Learningcited by 350
  14. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  15. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  16. 2021Scaling Laws for Transfercited by 265
  17. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  18. 2019Evaluating Protein Transfer Learning with TAPEcited by 229
  19. 2015Semi-supervised Sequence Learningcited by 198
  20. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  21. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  22. 2019Fair Generative Modeling via Weak Supervisioncited by 114
  23. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  24. 2017One Model To Learn Them Allcited by 86
  25. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  26. 2017Learning with Latent Languagecited by 75
  27. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  28. 2020Imitating Interactive Intelligencecited by 55
  29. 2017DeepSF: deep convolutional neural network for mapping protein sequences to foldscited by 46
  30. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  31. 2018On Self Modulation for Generative Adversarial Networkscited by 33
  32. 2020Emergent Communication Pretraining for Few-Shot Machine Translationcited by 13
  33. 2020Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 58; 2023: 43; 2024: 17; 2025: 12; 2026: 5.

162 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2024The Platonic Representation Hypothesiscited by 362
  10. 2021Choose a Transformer: Fourier or Galerkincited by 333
  11. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  12. 2023ClimaX: A foundation model for weather and climatecited by 325
38 more
  1. 2023One for All: Towards Training One Graph Model for All Classification Taskscited by 239
  2. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  3. 2022Multi-Game Decision Transformerscited by 212
  4. 2023Large Language Models as General Pattern Machinescited by 196
  5. 2022Online Decision Transformercited by 194
  6. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  7. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  8. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  9. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 156
  10. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  11. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  12. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  13. 2022Linearly Mapping from Image to Text Spacecited by 139
  14. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 139
  15. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  16. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  17. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  18. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 121
  19. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  20. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  21. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 109
  22. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  23. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  24. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  25. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  26. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  27. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  28. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  29. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  30. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  31. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  32. 2022Video Transformers: A Surveycited by 63
  33. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63
  34. 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
  35. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  36. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  37. 2024LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Promptingcited by 56
  38. 2023A Survey on Transformers in Reinforcement Learningcited by 50

50 most cited of 162 All 162 (JSON)