mlsanityData to 14 Oct 2026

Context-Aware Multimodal Pretraining

arXiv 2411.15099 ↗First posted 22 Nov 2024cs.CV cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Large-scale multimodal representation learning successfully optimizes for zero-shot transfer at test time. Yet the standard pretraining paradigm (contrastive learning on large amounts of image-text data) does not explicitly encourage representations to support few-shot adaptation. In this work, we propose a simple, but carefully designed extension to multimodal pretraining which enables representations to accommodate additional context. Using this objective, we show that vision-language models can be trained to exhibit significantly increased few-shot adaptation: across 21 downstream tasks, we find up to four-fold improvements in test-time sample efficiency, and average few-shot adaptation gains of over 5%, while retaining zero-shot generalization performance across model scales and training durations. In particular, equipped with simple, training-free, metric-based adaptation mechanisms, our representations easily surpass more complex and expensive optimization-based schemes, vastly simplifying generalization to new domains.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 91 of its 113 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

91 of 113 references.

References · 91

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  5. 2016Matching Networks for One Shot Learningcited by 2,255
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  11. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  13. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  14. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  15. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  16. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  17. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  18. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 776
  19. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  20. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  21. 2019Meta-Learning with Implicit Gradientscited by 525
  22. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  23. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  24. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  25. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  26. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  27. 2022Visual Classification via Description from Large Language Modelscited by 336
  28. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  29. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  30. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  31. 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 195
  32. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  33. 2020Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learningcited by 179
  34. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  35. 2015An open access repository of images on plant health to enable the development of mobile disease diagnosticscited by 135
  36. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  37. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  38. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 119

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  2. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  3. 2025Does CLIP perceive art the same way we do?cited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  2. 2023On the Performance of Multimodal Language Models
  3. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
  4. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
  5. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models
  6. 2025Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding
  7. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
  8. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering