mlsanityData to 7 Oct 2026

Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners

arXiv 2303.02151 ↗First posted 3 Mar 2023cs.CV cs.CLcited by 152

Not on the map. Why?

Cited by 62 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. We then question, if the more diverse pre-training knowledge can be cascaded to further assist few-shot representation learning. In this paper, we propose CaFo, a Cascade of Foundation models that incorporates diverse prior knowledge of various pre-training paradigms for better few-shot learning. Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge. Specifically, CaFo works by 'Prompt, Generate, then Cache'. Firstly, we leverage GPT-3 to produce textual inputs for prompting CLIP with rich downstream linguistic semantics. Then, we generate synthetic images via DALL-E to expand the few-shot training data without any manpower. At last, we introduce a learnable cache model to adaptively blend the predictions from CLIP and DINO. By such collaboration, CaFo can fully unleash the potential of different pre-training methods and unify them to perform state-of-the-art for few-shot classification. Code is available at https://github.com/ZrrSkywalker/CaFo.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). Builds on 64 of its 87 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

64 of 87 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2016Matching Networks for One Shot Learningcited by 2,246
  12. 2013Describing Textures in the Wildcited by 2,110
  13. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  14. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  15. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  16. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  17. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  18. 2019Natural Adversarial Examplescited by 1,401
  19. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  20. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  21. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  22. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  23. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  24. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  25. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  26. 2017Focal Loss for Dense Object Detectioncited by 749
  27. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  28. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  29. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  30. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  31. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  32. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  33. 2018Rethinking ImageNet Pre-trainingcited by 376
  34. 2022Visual Classification via Description from Large Language Modelscited by 331
  35. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  36. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  37. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  38. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 47; 2024: 62; 2025: 35; 2026: 8.

152 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  6. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  8. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  9. 2023Personalize Segment Anything Model with One Shotcited by 250
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  11. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  12. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
38 more
  1. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  2. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  3. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  4. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  5. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  6. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  7. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  8. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  9. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  10. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  11. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  12. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  13. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  14. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  15. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  16. 2023GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPTcited by 40
  17. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  18. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  19. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  20. 2024Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detectioncited by 35
  21. 2023Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysiscited by 34
  22. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Modelcited by 31
  23. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  24. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  25. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  26. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  27. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 24
  28. 2023Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Modelscited by 23
  29. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learningcited by 22
  30. 2023FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detectioncited by 19
  31. 2024No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentationcited by 18
  32. 2024Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentationcited by 17
  33. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Modelscited by 17
  34. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  35. 2024Parameter-efficient Prompt Learning for 3D Point Cloud Understandingcited by 16
  36. 2024EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splattingcited by 16
  37. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  38. 2024Prompt Categories Cluster for Weakly Supervised Semantic Segmentationcited by 16

50 most cited of 152 All 152 (JSON)