mlsanityData to 7 Oct 2026

Patching open-vocabulary models by interpolating weights

arXiv 2208.05592 ↗First posted 10 Aug 2022cs.CV cs.LGcited by 187

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optimal. We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate. Towards this goal, we introduce PAINT, a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched. On nine tasks where zero-shot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model. PAINT also allows a single model to be patched on multiple tasks and improves with model scale. Furthermore, we identify cases of broad transfer, where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes. Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on CLIP. Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 66 of its 96 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

66 of 96 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2016Learning without Forgettingcited by 2,465
  2. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  13. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  14. 2017Continual Learning with Deep Generative Replaycited by 1,152
  15. 2018Experience Replay for Continual Learningcited by 1,074
  16. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  17. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  18. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  19. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  20. 2021Robust fine-tuning of zero-shot modelscited by 785
  21. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  22. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  23. 2018On Lazy Training in Differentiable Programmingcited by 671
  24. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  25. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  26. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  27. 2021Editing Factual Knowledge in Language Modelscited by 548
  28. 2021Fast Model Editing at Scalecited by 491
  29. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  30. 2020What is being transferred in transfer learning?cited by 365
  31. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  32. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  33. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  34. 2019Continual learning with hypernetworkscited by 267
  35. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  36. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  37. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  38. 2020Supermasks in Superpositioncited by 223

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 53; 2024: 62; 2025: 46; 2026: 17.

187 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  4. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  6. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  10. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  11. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  12. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
38 more
  1. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  2. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  3. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  4. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  5. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  6. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  7. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  8. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  9. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  10. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  11. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  12. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  13. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 90
  14. 2023Deep Model Fusion: A Surveycited by 89
  15. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  16. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  17. 2024Parameter Competition Balancing for Model Mergingcited by 71
  18. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  19. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  20. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  21. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 54
  22. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  23. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  24. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  25. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  26. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  27. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  28. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  29. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  30. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  31. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  32. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 35
  33. 2024TaxaBind: A Unified Embedding Space for Ecological Applicationscited by 33
  34. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
  35. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
  36. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  37. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  38. 2024Robust Concept Erasure Using Task Vectorscited by 27

50 most cited of 187 All 187 (JSON)