mlsanityData to 14 Oct 2026

Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models

arXiv 2407.16526 ↗First posted 23 Jul 2024cs.CV cs.AI cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision language models (VLMs) demonstrate impressive capabilities in visual question answering and image captioning, acting as a crucial link between visual and language models. However, existing open-source VLMs heavily rely on pretrained and frozen vision encoders (such as CLIP). Despite CLIP's robustness across diverse domains, it still exhibits non-negligible image understanding errors. These errors propagate to the VLM responses, resulting in sub-optimal performance. In our work, we propose an efficient and robust method for updating vision encoders within VLMs. Our approach selectively and locally updates encoders, leading to substantial performance improvements on data where previous mistakes occurred, while maintaining overall robustness. Furthermore, we demonstrate the effectiveness of our method during continual few-shot updates. Theoretical grounding, generality, and computational efficiency characterize our approach.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 35 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

27 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions20 Feb
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2022Editing Models with Task Arithmeticcited by 1,120
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
13 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  3. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  4. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  5. 2022Visual Spatial Reasoningcited by 293
  6. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  7. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  8. 2023First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learningcited by 46
  9. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  10. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  11. 2022FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classificationcited by 26
  12. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  13. 2023XIMAGENET-12: An Explainable AI Benchmark Dataset for Model Robustness Evaluationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysiscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Encoder-Free Vision-Language Models
  2. 2025Efficient Few-Shot Continual Learning in Vision-Language Models
  3. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  5. 2024PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter
  6. 2025Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
  7. 2025A Survey on Efficient Vision-Language Models
  8. 2024BRAVE: Broadening the visual encoding of vision-language models