mlsanityData to 7 Oct 2026

Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT

arXiv 2305.00201 ↗First posted 29 Apr 2023cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Prompts have been proven to play a crucial role in large language models, and in recent years, vision models have also been using prompts to improve scalability for multiple downstream tasks. In this paper, we focus on adapting prompt design based on instruction tuning into a visual transformer model for image classification which we called Instruction-ViT. The key idea is to implement multi-modal prompts (text or image prompt) related to category information to guide the fine-tuning of the model. Based on the experiments of several image captionining tasks, the performance and domain adaptability were improved. Our work provided an innovative strategy to fuse multi-modal prompts with better performance and faster adaptability for visual classification models.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 29 Apr 2023.

Builds on

36 of 48 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
24 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  13. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  14. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  15. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  16. 2021Going deeper with Image Transformerscited by 759
  17. 2023Instruction Tuning with GPT-4cited by 676
  18. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  19. 2022Visual Prompting via Image Inpaintingcited by 248
  20. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  21. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  22. 2023When Brain-inspired AI Meets AGIcited by 54
  23. 2016Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusioncited by 23
  24. 2023Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuningcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 10; 2024: 3; 2025: 1.

14 papers.

Sort:

By area:

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  3. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  4. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  5. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  6. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  7. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  8. 2023Multimodality of AI for Education: Towards Artificial General Intelligencecited by 21
  9. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  10. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  11. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15
  12. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 10
2 more
  1. 2023SAM for Poultry Sciencecited by 6
  2. 2025Large Language Models for Bioinformaticscited by 3