mlsanityData to 7 Oct 2026

Language Models are General-Purpose Interfaces

arXiv 2206.06336 ↗First posted 13 Jun 2022cs.CLcited by 96

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Foundation models have received much attention due to their effectiveness across a broad range of downstream applications. Though there is a big convergence in terms of architecture, most pretrained models are typically still developed for specific tasks or modalities. In this work, we propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer. We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders. We subsume the advantages and capabilities from both causal and non-causal modeling, thereby combining the best of two worlds. Specifically, the proposed method not only inherits the capabilities of in-context learning and open-ended generation from causal language modeling, but also is conducive to finetuning because of the bidirectional encoders. More importantly, our approach seamlessly unlocks the combinations of the above capabilities, e.g., enabling in-context learning or instruction following with finetuned encoders. Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Builds on 87 of its 139 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

87 of 139 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  14. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  15. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  16. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  17. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  18. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  19. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  20. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  21. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  22. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  23. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  24. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  25. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  26. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  27. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  28. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  29. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  30. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  31. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  32. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  33. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  34. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  35. 2018Neural Network Acceptability Judgmentscited by 952
  36. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  37. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  38. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 43; 2024: 25; 2025: 7; 2026: 5.

96 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  5. 2022A Survey on In-context Learningcited by 717
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  7. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  9. 2024Large Language Models: A Surveycited by 506
  10. 2023Augmented Language Models: a Surveycited by 398
  11. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  12. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
38 more
  1. 2023Tool Learning with Foundation Modelscited by 302
  2. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  3. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  4. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  5. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  6. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  7. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  8. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  9. 2022Linearly Mapping from Image to Text Spacecited by 139
  10. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  13. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  14. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  15. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  16. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  17. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  18. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  19. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  20. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 88
  21. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  22. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  23. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  24. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  25. 2024Embodied Understanding of Driving Scenarioscited by 63
  26. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  27. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  28. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  29. 2024ARC: A Generalist Graph Anomaly Detector with In-Context Learningcited by 46
  30. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 45
  31. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  32. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  33. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  34. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  35. 2024Private Attribute Inference from Images with Vision-Language Modelscited by 35
  36. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  37. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  38. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27

50 most cited of 96 All 96 (JSON)