mlsanityData to 7 Oct 2026

Prismer: A Vision-Language Model with Multi-Task Experts

arXiv 2303.02506 ↗First posted 4 Mar 2023cs.LG cs.AI cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent vision-language models have shown impressive multi-modal generation capabilities. However, typically they require training huge models on massive datasets. As a more scalable alternative, we introduce Prismer, a data- and parameter-efficient vision-language model that leverages an ensemble of task-specific experts. Prismer only requires training of a small number of components, with the majority of network weights inherited from multiple readily-available, pre-trained experts, and kept frozen during training. By leveraging experts from a wide range of domains, we show Prismer can efficiently pool this expert knowledge and adapt it to various vision-language reasoning tasks. In our experiments, we show that Prismer achieves fine-tuned and few-shot learning performance which is competitive with current state-of-the-arts, whilst requiring up to two orders of magnitude less training data. Code is available at https://github.com/NVlabs/prismer.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 103 references that are in the map. First posted on arXiv 4 Mar 2023.

Builds on

86 of 103 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation12 Mar
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  4. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2021Vision Transformers for Dense Predictioncited by 1,789
  13. 2019Towards VQA Models That Can Readcited by 1,696
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  19. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  22. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  23. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  24. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  25. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  26. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  27. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  28. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  29. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  30. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  31. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  32. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  33. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  34. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  35. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  36. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  37. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  38. 2016Cross-stitch Networks for Multi-task Learningcited by 466

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 10; 2024: 8; 2025: 10.

28 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  3. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 122
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  5. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  8. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  9. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  10. 2023Structural Information Guided Multimodal Pre-training for Vehicle-centric Perceptioncited by 7
  11. 2023GRID: A Platform for General Robot Intelligence Developmentcited by 6
  12. 2024ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoningcited by 6
16 more
  1. 2024On Moving Object Segmentation from Monocular Video with Transformerscited by 6
  2. 2025MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregationcited by 6
  3. 2023Manipulating the Label Space for In-Context Classificationcited by 5
  4. 2025ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Expertscited by 5
  5. 2025Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoderscited by 5
  6. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  7. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 4
  8. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractorcited by 3
  9. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  10. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancementcited by 2
  11. 2025Scope: Selective Cross-modal Orchestration of Visual Perception Expertscited by 2
  12. 2023Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perceptioncited by 1
  13. 2024Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Modelcited by 1
  14. 2023Apollo: Zero-shot MultiModal Reasoning with Multiple Expertscited by 0
  15. 2025VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Modelscited by 0
  16. 2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  3. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  4. 2025Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  6. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
  7. 2026Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents
  8. 2022Localized Latent Updates for Fine-Tuning Vision-Language Models