mlsanityData to 7 Oct 2026

Review of Large Vision Models and Visual Prompt Engineering

arXiv 2307.00855 ↗First posted 3 Jul 2023cs.CV cs.AIcited by 77

Not on the map. Why?

Abstract (arXiv)

Visual prompt engineering is a fundamental technology in the field of visual and image Artificial General Intelligence, serving as a key component for achieving zero-shot capabilities. As the development of large vision models progresses, the importance of prompt engineering becomes increasingly evident. Designing suitable prompts for specific visual tasks has emerged as a meaningful research direction. This review aims to summarize the methods employed in the computer vision domain for large vision models and visual prompt engineering, exploring the latest advancements in visual prompt engineering. We present influential large models in the visual domain and a range of prompt engineering methods employed on these models. It is our hope that this review provides a comprehensive and systematic description of prompt engineering methods based on large visual models, offering valuable insights for future researchers in their exploration of this field.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 129 of its 173 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

129 of 173 references.

References · 129

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2023Segment Anythingcited by 8,158
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  15. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  16. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  18. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  19. 2021Grounded Language-Image Pre-trainingcited by 1,147
  20. 2021Transformers in Vision: A Surveycited by 1,009
  21. 2022Diffusion Models in Vision: A Surveycited by 1,008
  22. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  23. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  24. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  25. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  26. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  27. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  28. 2021Transformer in Transformercited by 632
  29. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  30. 2023Segment Everything Everywhere All at Oncecited by 546
  31. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  32. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  33. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  34. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  35. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  36. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  37. 2022Decoupled Knowledge Distillationcited by 380
  38. 2017One-Shot Learning for Semantic Segmentationcited by 337

50 most cited of 129 All 129 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 32; 2025: 25; 2026: 5.

77 papers.

Sort:

By area:

  1. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  2. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
  3. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  4. 2025HVI: A New Color Space for Low-light Image Enhancementcited by 62
  5. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  7. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  8. 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
  9. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  10. 2023GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPTcited by 40
  11. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
  12. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
38 more
  1. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  2. 2023Multimodality of AI for Education: Towards Artificial General Intelligencecited by 21
  3. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  4. 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysiscited by 20
  5. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  6. 2023Self-Sampling Meta SAM: Enhancing Few-shot Medical Image Segmentation with Meta-Learningcited by 18
  7. 2023RadOnc-GPT: A Large Language Model for Radiation Oncologycited by 18
  8. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  9. 2024Generative AI Literacy: Twelve Defining Competenciescited by 13
  10. 2023Evaluating Large Language Models for Radiology Natural Language Processingcited by 12
  11. 2023More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problemscited by 10
  12. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  13. 2025CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillationcited by 10
  14. 2023Ophtha-LLaMA2: A Large Language Model for Ophthalmologycited by 9
  15. 2024HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenizationcited by 8
  16. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  17. 2024Automated User Story Generation with Test Case Specification Using Large Language Modelcited by 7
  18. 2024Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMscited by 6
  19. 2023Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classificationcited by 5
  20. 2023Adaptive Human Trajectory Prediction via Latent Corridorscited by 5
  21. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  22. 2025Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Taskscited by 5
  23. 2024SCAPE: Searching Conceptual Architecture Prompts using Evolutioncited by 4
  24. 2024Bioinformatics and Biomedical Informatics with ChatGPT: Year One Reviewcited by 4
  25. 2024FocSAM: Delving Deeply into Focused Objects in Segmenting Anythingcited by 4
  26. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compressioncited by 4
  27. 2024Legal Evalutions and Challenges of Large Language Modelscited by 4
  28. 2024The Prompt Canvas: A Literature-Based Practitioner Guide for Creating Effective Prompts in Large Language Modelscited by 4
  29. 2025LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systemscited by 4
  30. 2023PharmacyGPT: The AI Pharmacistcited by 3
  31. 2024A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligencecited by 3
  32. 2024Evaluating the Efficacy of Prompt-Engineered Large Multimodal Models Versus Fine-Tuned Vision Transformers in Image-Based Security Applicationscited by 3
  33. 2024PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classificationcited by 3
  34. 2024Multimodal Contrastive In-Context Learningcited by 3
  35. 2024Stability Analysis of ChatGPT-based Sentiment Analysis in AI Quality Assurancecited by 2
  36. 2024Promptly Yours? A Human Subject Study on Prompt Inference in AI-Generated Artcited by 2
  37. 2025GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRIcited by 2
  38. 2025Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognitioncited by 2

50 most cited of 77 All 77 (JSON)